Sep 16 18:45:40.891126 ip-10-0-142-130 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:41.350960 ip-10-0-142-130 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:41.350960 ip-10-0-142-130 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:41.350960 ip-10-0-142-130 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:41.350960 ip-10-0-142-130 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:41.350960 ip-10-0-142-130 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:41.353361 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.353278    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:41.357099 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357083    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.357099 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357099    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357102    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357105    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357107    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357110    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357113    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357117    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357119    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357122    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357127    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357131    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357134    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357137    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357140    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357142    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357145    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357148    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357151    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357153    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.357161 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357155    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357159    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357161    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357164    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357166    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357169    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357172    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357175    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357178    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357180    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357183    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357185    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357188    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357190    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357193    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357196    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357198    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357201    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357203    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357207    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.357608 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357209    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357212    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357214    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357216    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357219    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357222    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357226    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357230    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357232    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357235    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357238    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357241    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357244    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357246    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357250    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357253    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357255    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357258    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357261    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.358100 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357263    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357266    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357269    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357271    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357274    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357276    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357279    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357282    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357284    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357287    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357289    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357292    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357295    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357297    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357299    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357302    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357304    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357307    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357309    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357311    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.358578 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357314    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357317    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357319    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357321    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357324    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357326    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357329    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357333    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357336    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357338    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357341    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357343    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357346    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357348    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357351    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357355    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357358    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357360    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357363    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.359076 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.357367    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358063    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358071    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358074    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358078    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358082    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358086    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358090    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358093    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358095    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358100    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358104    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358107    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358110    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358113    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358116    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358119    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358122    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358125    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.359570 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358128    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358132    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358135    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358138    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358141    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358143    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358146    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358149    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358151    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358154    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358157    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358159    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358162    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358165    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358167    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358171    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358173    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358176    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358179    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358182    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.360057 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358184    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358187    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358190    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358192    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358195    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358197    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358200    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358203    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358205    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358208    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358211    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358213    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358216    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358218    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358221    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358224    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358227    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358230    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358232    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358235    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.360538 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358237    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358239    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358243    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358246    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358249    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358252    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358255    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358258    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358261    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358264    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358266    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358269    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358272    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358274    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358277    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358279    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358283    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358290    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358293    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.361127 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358295    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358298    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358300    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358303    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358305    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358308    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358310    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358313    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358315    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358318    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358323    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358325    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358328    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358330    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358332    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358335    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358337    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358340    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358342    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.361605 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358345    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358347    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358699    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358703    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358707    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358710    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358713    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358716    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358718    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358721    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358725    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358728    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358731    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358733    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358736    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358739    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358742    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358744    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358747    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358749    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.362115 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358752    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358754    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358757    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358760    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358762    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358764    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358767    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358769    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358773    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358776    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358780    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358784    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358787    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358790    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358793    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358796    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358798    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358801    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358803    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.362611 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358806    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358808    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358811    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358815    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358818    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358821    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358823    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358826    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358830    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358833    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358835    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358838    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358840    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358843    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358860    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358863    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358866    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358868    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.363078 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358871    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358873    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358876    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358879    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358881    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358884    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358886    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358889    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358891    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358894    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358896    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358898    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358902    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358904    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358907    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358910    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358914    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358917    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358920    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358923    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.363549 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358926    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358929    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358931    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358934    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358936    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358939    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358941    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358944    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358946    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358949    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358951    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358954    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358956    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358959    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358961    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358964    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358966    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358968    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358971    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358973    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.364094 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358976    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358978    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.358980    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359053    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359085    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359091    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359096    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359100    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359106    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359110    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359114    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359117    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359120    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359124    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359127    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359130    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359133    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359136    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359139    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359142    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359145    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359150    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359153    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359156    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:41.364584 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359159    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359162    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359166    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359169    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359172    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359175    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359178    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359181    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359184    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359186    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359189    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359193    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359196    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359199    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359202    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359205    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359210    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359215    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359217    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359220    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359223    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359226    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359230    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359233    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359236    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:41.365180 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359239    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359242    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359245    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359248    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359250    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359253    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359256    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359259    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359263    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359266    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359269    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359272    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359275    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359278    2578 flags.go:64] FLAG: --help="false"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359281    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359284    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359287    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359290    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359293    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359296    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359299    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359302    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359307    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:41.365781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359311    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359314    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359317    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359320    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359323    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359326    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359329    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359332    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359335    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359338    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359340    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359343    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359348    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359351    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359354    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359356    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359359    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359362    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359365    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359368    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359372    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359375    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359379    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359381    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359384    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:41.366376 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359387    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359390    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359392    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359395    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359398    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359405    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359408    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359415    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359418    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359421    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359425    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359428    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359431    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359434    2578 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359437    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359440    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0868729162248a4b3"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359444    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359446    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359449    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359452    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359456    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359458    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359461    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359464    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359467    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:41.367329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359470    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359474    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359476    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359479    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359482    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359485    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359487    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359490    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359493    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359496    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359499    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359502    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359505    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359507    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359512    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359516    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359519    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359522    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359524    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359529    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359532    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359535    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359539    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359542    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359545    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:41.368463 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359548    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359551    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359553    2578 flags.go:64] FLAG: --v="2"
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359557    2578 flags.go:64] FLAG: --version="false"
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359561    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359564    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.359567    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359677    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359681    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359684    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359687    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359689    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359691    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359694    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359697    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359699    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359701    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359704    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359706    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359709    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359712    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.369476 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359716    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359720    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359723    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359726    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359728    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359731    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359733    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359736    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359738    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359741    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359743    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359746    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359748    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359751    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359754    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359756    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359759    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359762    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359764    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.370028 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359767    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359770    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359772    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359775    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359777    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359780    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359782    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359785    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359787    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359790    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359792    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359794    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359797    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359801    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359805    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359808    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359811    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359814    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359816    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359819    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.370507 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359821    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359825    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359831    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359834    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359837    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359839    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359842    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359844    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359861    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359864    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359866    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359869    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359871    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359874    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359876    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359879    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359882    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359884    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.371013 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359887    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359889    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359892    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359894    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359898    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359901    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359905    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359909    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359912    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359914    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359918    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359920    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359923    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359927    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359930    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359933    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359936    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359939    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359942    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359944    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.371480 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359947    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359950    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359953    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359956    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359959    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359961    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.359964    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.360635    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.370480    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.370495    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.371247    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.371255    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371312    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371316    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371320    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.371981 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371323    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371326    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371329    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371332    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371337    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371341    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371344    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371347    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371350    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371353    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371356    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371359    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371362    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371365    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371368    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371370    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371373    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371376    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371379    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.372400 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371382    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371384    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371387    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371390    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371392    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371395    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371398    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371401    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371404    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371406    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371409    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371412    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371415    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371418    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371421    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371423    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371426    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371429    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371431    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371433    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.373004 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371436    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371439    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371441    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371444    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371446    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371450    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371453    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371457    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371459    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371462    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371465    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371468    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371470    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371473    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371475    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371478    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371480    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371483    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371486    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.373488 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371489    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371492    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371495    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371498    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371500    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371502    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371505    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371508    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371510    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371513    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371515    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371518    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371520    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371523    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371525    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371528    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371531    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371533    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371537    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371539    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.373979 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371542    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371544    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371547    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371550    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371552    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371555    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371558    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371560    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371563    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371566    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371568    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371571    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371574    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371577    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371581    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371586    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371589    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.374463 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371591    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.371597    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371710    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371715    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371718    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371721    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371724    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371726    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371729    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371732    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371736    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371740    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371743    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371746    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371750    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:41.374886 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371753    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371756    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371758    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371761    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371764    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371767    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371769    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371772    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371775    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371788    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371793    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371796    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371799    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371802    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371805    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371807    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371810    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371813    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371815    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371818    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:41.375272 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371820    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371823    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371826    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371831    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371834    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371837    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371840    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371842    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371845    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371861    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371865    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371870    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371873    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371877    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371880    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371883    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371886    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371889    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:41.375763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371891    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371894    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371896    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371899    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371902    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371904    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371907    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371909    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371912    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371914    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371917    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371920    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371922    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371925    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371927    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371930    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371932    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371934    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371937    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371940    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:41.376250 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371942    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371945    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371947    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371950    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371952    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371955    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371958    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371961    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371963    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371966    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371968    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371971    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371973    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371976    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371979    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371981    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371984    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371987    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371989    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371992    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:41.376720 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371995    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.371997    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.372000    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.372002    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.372005    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.372007    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:41.372010    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.372015    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.372121    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.374598    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.375623    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.375725    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:41.377227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.375765    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:41.378261 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.378248    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:41.380209 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.380193    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:41.392523 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.392505    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:41.395391 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.395378    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:41.396214 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.396188    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:41.399727 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.399702    2578 fs.go:135] Filesystem UUIDs: map[20fcb368-fe4c-497f-9ebb-eb1e51f12519:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 afff5c7c-7fee-4ad0-a40a-ed63ce667423:/dev/nvme0n1p4]
Sep 16 18:45:41.399775 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.399727    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:41.405343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.405134    2578 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:41.403463079 +0000 UTC m=+0.374940313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136356 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c013ebf1d1f9f3db9876b640f84d9 SystemUUID:ec2c013e-bf1d-1f9f-3db9-876b640f84d9 BootID:c96f1e46-e331-4e6a-b27b-24326f51aac8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:cd:7c:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:cd:7c:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:1b:50:bd:38:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:41.405343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.405333    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:41.405892 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.405875    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:41.409383 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.409344    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:41.409472 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.409402    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:41.409570 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.409389    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:41.410215 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.410203    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:41.410265 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.410221    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:41.410265 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.410233    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:41.411321 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.411309    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:41.412169 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.412157    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:41.415406 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415393    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:41.415542 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415531    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:41.415606 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415571    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:41.415606 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415585    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:41.415606 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415600    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:41.415721 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415613    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:41.415721 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.415625    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:41.416021 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.416007    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:41.416129 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.416032    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:41.418192 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.418173    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:41.420406 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.420392    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:41.420455 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.420423    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:41.421927 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421915    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421932    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421938    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421943    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421951    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421956    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421969    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421974    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421981    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:41.421984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421986    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:41.422210 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.421995    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:41.422210 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.422000    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:41.422882 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.422871    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:41.422882 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.422882    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:41.426785 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.426771    2578 server.go:1268] "Started kubelet"
Sep 16 18:45:41.426890 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.426843    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:41.427054 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.427000    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:41.427123 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.427084    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:41.427543 ip-10-0-142-130 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:41.428315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.428289    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:41.429051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.429031    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:41.429525 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.429507    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:41.429623 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.429604    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:41.429685 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.429639    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:41.432203 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.432182    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:41.434375 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.433403    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-130.ec2.internal.18d5e17a1657ae54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-130.ec2.internal,UID:ip-10-0-142-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-130.ec2.internal,},FirstTimestamp:2026-09-16 18:45:41.426744916 +0000 UTC m=+0.398222156,LastTimestamp:2026-09-16 18:45:41.426744916 +0000 UTC m=+0.398222156,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-130.ec2.internal,}"
Sep 16 18:45:41.435348 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.435332    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:41.436361 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.436348    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:41.436441 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.436427    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.436953 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.436933    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:41.437041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437025    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:41.437041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437035    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:41.437134 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437110    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:41.437134 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437117    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:41.437380 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437360    2578 factory.go:55] Registering systemd factory
Sep 16 18:45:41.437447 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437384    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:41.437591 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437580    2578 factory.go:153] Registering CRI-O factory
Sep 16 18:45:41.437641 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437593    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:41.437641 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437631    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:41.437708 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437647    2578 factory.go:103] Registering Raw factory
Sep 16 18:45:41.437708 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.437661    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:41.438210 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.438199    2578 manager.go:349] Starting recovery of all containers
Sep 16 18:45:41.441246 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.441219    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts99f"
Sep 16 18:45:41.441530 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.441500    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:41.441640 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.441613    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:41.449718 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.449655    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts99f"
Sep 16 18:45:41.450556 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.450541    2578 manager.go:354] Recovery completed
Sep 16 18:45:41.454382 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.454371    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.456162 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456145    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.456232 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456175    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.456232 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456185    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.456655 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456640    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:41.456655 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456653    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:41.456655 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456667    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:41.456834 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.456714    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.457397 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.457382    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.457468 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.457407    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.457468 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.457417    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.458082 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.458064    2578 policy_none.go:49] "None policy: Start"
Sep 16 18:45:41.458149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.458087    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:41.458149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.458102    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:41.458225 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.458164    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-130.ec2.internal.18d5e17a18188c11  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-130.ec2.internal,UID:ip-10-0-142-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-130.ec2.internal,},FirstTimestamp:2026-09-16 18:45:41.456161809 +0000 UTC m=+0.427639051,LastTimestamp:2026-09-16 18:45:41.456161809 +0000 UTC m=+0.427639051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-130.ec2.internal,}"
Sep 16 18:45:41.459546 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.459535    2578 policy_none.go:47] "Start"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494358    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.494414    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494427    2578 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494636    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494648    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494813    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494921    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.494930    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.495293    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:41.510951 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.495332    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.595454 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.595426    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.595994 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.595969    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:41.596352 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.596323    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.596633 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.596356    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.596633 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.596371    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.596633 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.596397    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.597366 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.597347    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:41.597450 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.597394    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:41.597450 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.597414    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:41.597542 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.597450    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:41.599163 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.599147    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:41.604810 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.604766    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.604810 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.604786    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-130.ec2.internal\": node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.621672 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.621649    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.698225 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.698188    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"]
Sep 16 18:45:41.698336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.698264    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.699811 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.699791    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.699910 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.699822    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.699910 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.699833    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.701328 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.701316    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.701452 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.701436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.701498 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.701467    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.702084 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702068    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.702161 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702094    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.702161 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702105    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.702161 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702074    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.702269 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702166    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.702269 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.702179    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.703340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.703325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.703401 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.703351    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:41.704015 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.703999    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:41.704097 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.704021    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:41.704097 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.704032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:41.722656 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.722634    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.734505 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.734483    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-130.ec2.internal\" not found" node="ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.738669 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.738653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.738734 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.738675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.738734 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.738681    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-130.ec2.internal\" not found" node="ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.738734 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.738694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7e51c5c4cf47eed84f85b77e7cf2641-config\") pod \"kube-apiserver-proxy-ip-10-0-142-130.ec2.internal\" (UID: \"d7e51c5c4cf47eed84f85b77e7cf2641\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.823147 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.823121    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:41.839834 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.839950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7e51c5c4cf47eed84f85b77e7cf2641-config\") pod \"kube-apiserver-proxy-ip-10-0-142-130.ec2.internal\" (UID: \"d7e51c5c4cf47eed84f85b77e7cf2641\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.839950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.839950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.839950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7e51c5c4cf47eed84f85b77e7cf2641-config\") pod \"kube-apiserver-proxy-ip-10-0-142-130.ec2.internal\" (UID: \"d7e51c5c4cf47eed84f85b77e7cf2641\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.839950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:41.839903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b3fe0053cfc1761b28dc935cdabb16d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal\" (UID: \"4b3fe0053cfc1761b28dc935cdabb16d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:41.923419 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:41.923351    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.023889 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.023833    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.037009 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.036991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:42.043056 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.042507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:42.125103 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.124956    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.225580 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.225502    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.326091 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.326064    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.375506 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.375480    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:42.375902 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.375633    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:42.426972 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.426948    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.453669 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.453632    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:42 +0000 UTC" deadline="2028-06-02 07:44:57.680113812 +0000 UTC"
Sep 16 18:45:42.453669 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.453662    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14988h59m15.226455837s"
Sep 16 18:45:42.527954 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.527922    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.557839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.557817    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:42.614637 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:42.614607    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e51c5c4cf47eed84f85b77e7cf2641.slice/crio-64797b4697a4537f31a3c7e6576ad8c2f54dee9a25ed69b6a6f8da420046a997 WatchSource:0}: Error finding container 64797b4697a4537f31a3c7e6576ad8c2f54dee9a25ed69b6a6f8da420046a997: Status 404 returned error can't find the container with id 64797b4697a4537f31a3c7e6576ad8c2f54dee9a25ed69b6a6f8da420046a997
Sep 16 18:45:42.620485 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.620469    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:42.628681 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.628664    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.629027 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:42.629008    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3fe0053cfc1761b28dc935cdabb16d.slice/crio-d91ae416a918bb959e44c3a4cd8df8ae110f58e9efe26750c2d6a91023a3a3eb WatchSource:0}: Error finding container d91ae416a918bb959e44c3a4cd8df8ae110f58e9efe26750c2d6a91023a3a3eb: Status 404 returned error can't find the container with id d91ae416a918bb959e44c3a4cd8df8ae110f58e9efe26750c2d6a91023a3a3eb
Sep 16 18:45:42.706206 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.706185    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:42.729230 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:42.729211    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-130.ec2.internal\" not found"
Sep 16 18:45:42.781315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.781253    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:42.837127 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.837110    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:42.851893 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.851874    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:42.853195 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.853182    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal"
Sep 16 18:45:42.865856 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:42.865840    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:43.416135 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.416086    2578 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:43.427565 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.427252    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:43.429449 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.429417    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wc6zk","openshift-ovn-kubernetes/ovnkube-node-5xh5m","kube-system/global-pull-secret-syncer-mwm6b","kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b","openshift-image-registry/node-ca-52748","openshift-multus/network-metrics-daemon-zpmsl","openshift-network-diagnostics/network-check-target-xsm6d","openshift-network-operator/iptables-alerter-vpqf8","kube-system/konnectivity-agent-crttb","openshift-cluster-node-tuning-operator/tuned-lvsmf","openshift-dns/node-resolver-d979r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal","openshift-multus/multus-55hqc"]
Sep 16 18:45:43.431421 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.431394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.433343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.433322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.435487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.434893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.435487 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.434965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:43.435487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.435042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:43.435487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.435269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:43.435487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.435278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vjtvs\""
Sep 16 18:45:43.435728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.435678    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:43.436205 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.436181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.436420 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.436404    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.436513 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.436404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6gmd7\""
Sep 16 18:45:43.436677 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.436665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.439158 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.439138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.439577 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.439521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b4qcn\""
Sep 16 18:45:43.440651 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.439815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:43.440651 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.440122    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:43.440651 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.440138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.440651 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.440367    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.441752 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.441733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.442747 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.442721    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.442877 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.442857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.444219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.445047    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.445301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.445322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.445374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h6nrr\""
Sep 16 18:45:43.446107 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.445661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kq9zl\""
Sep 16 18:45:43.447443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.447050    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.447538 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.447464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:43.447629 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.447603    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:43.448238 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.448214    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.449489 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.449472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.449790 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.449769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.450065 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-system-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-kubelet\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-sys-fs\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.450149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysconfig\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-conf\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-run\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxv49\" (UniqueName: \"kubernetes.io/projected/c614891f-8a28-49d4-994f-30fc9308cfaf-kube-api-access-kxv49\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-dbus\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-bin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-multus-certs\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-kubernetes\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-var-lib-kubelet\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-host\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-etc-kubernetes\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvcb\" (UniqueName: \"kubernetes.io/projected/e2554090-4b49-4158-9277-23fd3dbe8ac7-kube-api-access-xsvcb\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-etc-selinux\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-lib-modules\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-cnibin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-cni-binary-copy\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-hostroot\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-conf-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-socket-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-device-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:43.450831 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f35ea27-56a7-4718-9e06-4fbfdb490581-konnectivity-ca\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-modprobe-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-tmp\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.450985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-socket-dir-parent\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-k8s-cni-cncf-io\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-netns\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-daemon-config\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-systemd\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-kubelet-config\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-multus\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-registration-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6x8l\" (UniqueName: \"kubernetes.io/projected/82abb29f-3294-46b2-b81f-41d42579b761-kube-api-access-w6x8l\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-74qwp\""
Sep 16 18:45:43.451427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f35ea27-56a7-4718-9e06-4fbfdb490581-agent-certs\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.452000 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-sys\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.452000 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-tuned\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.452000 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.451629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.453089 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.453014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.453253 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.453225    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:43.456476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.456458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.458526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.458510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.458828 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.458807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zk2hq\""
Sep 16 18:45:43.459290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.458713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:43.459290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.459025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.459290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.458715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:43.459290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.459260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:43.459638 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.458671    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:43.460723 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.460629    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:43.461068 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.460961    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.461424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.461164    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.461505 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.461463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jgp7k\""
Sep 16 18:45:43.461894 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.461749    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-88f4q\""
Sep 16 18:45:43.461894 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.461772    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:43.463151 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.462997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:43.527124 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.527090    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-25zg6"
Sep 16 18:45:43.539916 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.538596    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:43.539916 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.538812    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-25zg6"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.552888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-kubelet-config\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.552934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-host\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.552965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4hw\" (UniqueName: \"kubernetes.io/projected/6750ad10-4915-4413-a651-f22681422251-kube-api-access-bn4hw\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.552991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9j56\" (UniqueName: \"kubernetes.io/projected/e86c725d-a1ce-429e-8073-83919fcff64b-kube-api-access-p9j56\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.552983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-kubelet-config\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f35ea27-56a7-4718-9e06-4fbfdb490581-agent-certs\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-tuned\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6750ad10-4915-4413-a651-f22681422251-iptables-alerter-script\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-sys-fs\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-env-overrides\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q67g\" (UniqueName: \"kubernetes.io/projected/2b7e81bb-1f13-4f36-a344-952e6946dd68-kube-api-access-9q67g\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-hosts-file\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-run\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.553315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxv49\" (UniqueName: \"kubernetes.io/projected/c614891f-8a28-49d4-994f-30fc9308cfaf-kube-api-access-kxv49\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.554183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553360    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:43.554183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-sys-fs\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.554183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-run\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.554183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.553987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-dbus\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96514ca7-0140-4654-9b6c-7c16f636afd6-dbus\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-multus-certs\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvcb\" (UniqueName: \"kubernetes.io/projected/e2554090-4b49-4158-9277-23fd3dbe8ac7-kube-api-access-xsvcb\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-systemd-units\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6750ad10-4915-4413-a651-f22681422251-host-slash\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-var-lib-kubelet\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-host\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-etc-kubernetes\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-device-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-var-lib-kubelet\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-device-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f35ea27-56a7-4718-9e06-4fbfdb490581-konnectivity-ca\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.554975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-tmp\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.556327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7sf\" (UniqueName: \"kubernetes.io/projected/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-kube-api-access-hb7sf\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-netns\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-daemon-config\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-netns\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-var-lib-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-host\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-bin\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-multus-certs\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.555387    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.555466    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.055447972 +0000 UTC m=+3.026925224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-etc-kubernetes\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-netd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-tmp-dir\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-systemd\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-netns\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-multus\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.557341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-registration-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6x8l\" (UniqueName: \"kubernetes.io/projected/82abb29f-3294-46b2-b81f-41d42579b761-kube-api-access-w6x8l\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovn-node-metrics-cert\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-script-lib\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-serviceca\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.555998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-registration-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-systemd\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-multus\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-sys\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-log-socket\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-system-cni-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-system-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-kubelet\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-sys\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-systemd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.558292 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-etc-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-ovn\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-daemon-config\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-cnibin\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-system-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-kubelet\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f35ea27-56a7-4718-9e06-4fbfdb490581-konnectivity-ca\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysconfig\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-conf\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-bin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysconfig\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-var-lib-cni-bin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj7z\" (UniqueName: \"kubernetes.io/projected/a1a862c7-4d03-440d-a483-b0dfb7f21042-kube-api-access-tqj7z\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.559256 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-sysctl-conf\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-kubernetes\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-etc-selinux\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-kubelet\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-node-log\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-config\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-kubernetes\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-lib-modules\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-etc-selinux\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-cnibin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-cni-binary-copy\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-hostroot\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-conf-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-lib-modules\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-socket-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.560183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-hostroot\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-modprobe-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-socket-dir-parent\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9fh\" (UniqueName: \"kubernetes.io/projected/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-kube-api-access-fs9fh\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-cnibin\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-k8s-cni-cncf-io\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.556994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-conf-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-slash\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82abb29f-3294-46b2-b81f-41d42579b761-socket-dir\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-modprobe-d\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-socket-dir-parent\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-host-run-k8s-cni-cncf-io\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2554090-4b49-4158-9277-23fd3dbe8ac7-multus-cni-dir\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.557257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2554090-4b49-4158-9277-23fd3dbe8ac7-cni-binary-copy\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.561113 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.560982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-tmp\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.562804 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.562780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f35ea27-56a7-4718-9e06-4fbfdb490581-agent-certs\") pod \"konnectivity-agent-crttb\" (UID: \"9f35ea27-56a7-4718-9e06-4fbfdb490581\") " pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.564076 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.564050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxv49\" (UniqueName: \"kubernetes.io/projected/c614891f-8a28-49d4-994f-30fc9308cfaf-kube-api-access-kxv49\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.566949 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.565675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c614891f-8a28-49d4-994f-30fc9308cfaf-etc-tuned\") pod \"tuned-lvsmf\" (UID: \"c614891f-8a28-49d4-994f-30fc9308cfaf\") " pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.566949 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.566139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvcb\" (UniqueName: \"kubernetes.io/projected/e2554090-4b49-4158-9277-23fd3dbe8ac7-kube-api-access-xsvcb\") pod \"multus-55hqc\" (UID: \"e2554090-4b49-4158-9277-23fd3dbe8ac7\") " pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.567110 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.567025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6x8l\" (UniqueName: \"kubernetes.io/projected/82abb29f-3294-46b2-b81f-41d42579b761-kube-api-access-w6x8l\") pod \"aws-ebs-csi-driver-node-g888b\" (UID: \"82abb29f-3294-46b2-b81f-41d42579b761\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.602202 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.602138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal" event={"ID":"d7e51c5c4cf47eed84f85b77e7cf2641","Type":"ContainerStarted","Data":"64797b4697a4537f31a3c7e6576ad8c2f54dee9a25ed69b6a6f8da420046a997"}
Sep 16 18:45:43.603593 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.603563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal" event={"ID":"4b3fe0053cfc1761b28dc935cdabb16d","Type":"ContainerStarted","Data":"d91ae416a918bb959e44c3a4cd8df8ae110f58e9efe26750c2d6a91023a3a3eb"}
Sep 16 18:45:43.616353 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.616330    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:43.657300 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q67g\" (UniqueName: \"kubernetes.io/projected/2b7e81bb-1f13-4f36-a344-952e6946dd68-kube-api-access-9q67g\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-hosts-file\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-systemd-units\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6750ad10-4915-4413-a651-f22681422251-host-slash\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-hosts-file\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-systemd-units\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7sf\" (UniqueName: \"kubernetes.io/projected/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-kube-api-access-hb7sf\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6750ad10-4915-4413-a651-f22681422251-host-slash\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-netns\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-var-lib-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-bin\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-netd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-tmp-dir\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovn-node-metrics-cert\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-netns\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-var-lib-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-script-lib\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-serviceca\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-bin\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-log-socket\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-cni-netd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-system-cni-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.657833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-log-socket\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-system-cni-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-systemd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.657994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-etc-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-ovn\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-systemd\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-cnibin\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-etc-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-tmp-dir\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-cnibin\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-ovn\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-run-openvswitch\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.658653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj7z\" (UniqueName: \"kubernetes.io/projected/a1a862c7-4d03-440d-a483-b0dfb7f21042-kube-api-access-tqj7z\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-kubelet\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-node-log\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-serviceca\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-config\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-script-lib\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-kubelet\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9fh\" (UniqueName: \"kubernetes.io/projected/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-kube-api-access-fs9fh\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-slash\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-slash\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-node-log\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-host\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1a862c7-4d03-440d-a483-b0dfb7f21042-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.659443 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-host\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4hw\" (UniqueName: \"kubernetes.io/projected/6750ad10-4915-4413-a651-f22681422251-kube-api-access-bn4hw\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9j56\" (UniqueName: \"kubernetes.io/projected/e86c725d-a1ce-429e-8073-83919fcff64b-kube-api-access-p9j56\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b7e81bb-1f13-4f36-a344-952e6946dd68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6750ad10-4915-4413-a651-f22681422251-iptables-alerter-script\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-env-overrides\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.659056    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.659120    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.159096984 +0000 UTC m=+3.130574212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.659224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b7e81bb-1f13-4f36-a344-952e6946dd68-cni-binary-copy\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.658920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovnkube-config\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.659322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1a862c7-4d03-440d-a483-b0dfb7f21042-env-overrides\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.660125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.659405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6750ad10-4915-4413-a651-f22681422251-iptables-alerter-script\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.663437 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.660601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a862c7-4d03-440d-a483-b0dfb7f21042-ovn-node-metrics-cert\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.666112 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.666089    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:43.666112 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.666115    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:43.666287 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.666130    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:43.666287 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:43.666193    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.166174639 +0000 UTC m=+3.137651883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:43.669141 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.669111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q67g\" (UniqueName: \"kubernetes.io/projected/2b7e81bb-1f13-4f36-a344-952e6946dd68-kube-api-access-9q67g\") pod \"multus-additional-cni-plugins-wc6zk\" (UID: \"2b7e81bb-1f13-4f36-a344-952e6946dd68\") " pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.670747 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.670716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9fh\" (UniqueName: \"kubernetes.io/projected/cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b-kube-api-access-fs9fh\") pod \"node-resolver-d979r\" (UID: \"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b\") " pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:43.671153 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.671130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7sf\" (UniqueName: \"kubernetes.io/projected/9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069-kube-api-access-hb7sf\") pod \"node-ca-52748\" (UID: \"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069\") " pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.671834 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.671810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj7z\" (UniqueName: \"kubernetes.io/projected/a1a862c7-4d03-440d-a483-b0dfb7f21042-kube-api-access-tqj7z\") pod \"ovnkube-node-5xh5m\" (UID: \"a1a862c7-4d03-440d-a483-b0dfb7f21042\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.672254 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.672228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9j56\" (UniqueName: \"kubernetes.io/projected/e86c725d-a1ce-429e-8073-83919fcff64b-kube-api-access-p9j56\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:43.673365 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.673341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4hw\" (UniqueName: \"kubernetes.io/projected/6750ad10-4915-4413-a651-f22681422251-kube-api-access-bn4hw\") pod \"iptables-alerter-vpqf8\" (UID: \"6750ad10-4915-4413-a651-f22681422251\") " pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.746864 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.746811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:45:43.756312 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.756287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvsmf"
Sep 16 18:45:43.768817 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.768794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55hqc"
Sep 16 18:45:43.776783 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.776760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b"
Sep 16 18:45:43.784603 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.784573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-52748"
Sep 16 18:45:43.795195 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.795171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vpqf8"
Sep 16 18:45:43.809394 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.809376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:45:43.818447 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.818426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wc6zk"
Sep 16 18:45:43.826651 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:43.826632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d979r"
Sep 16 18:45:44.062017 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:44.061932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:44.062179 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.062108    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.062240 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.062184    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.062163057 +0000 UTC m=+4.033640289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.162320 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:44.162277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:44.162508 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.162402    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.162508 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.162471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.162448541 +0000 UTC m=+4.133925769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.263278 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:44.263161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:44.263454 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.263361    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:44.263454 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.263380    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:44.263454 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.263389    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.263454 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:44.263436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.263423576 +0000 UTC m=+4.234900805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.543338 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:44.543290    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:44 +0000 UTC" deadline="2028-06-21 01:54:13.909834788 +0000 UTC"
Sep 16 18:45:44.543338 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:44.543332    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15439h8m29.366506854s"
Sep 16 18:45:44.679529 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.679499    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82abb29f_3294_46b2_b81f_41d42579b761.slice/crio-0d4725e94795749c21e92f3e8c0b283cca9e7ab63a1ee6f7894cd65663d4ef21 WatchSource:0}: Error finding container 0d4725e94795749c21e92f3e8c0b283cca9e7ab63a1ee6f7894cd65663d4ef21: Status 404 returned error can't find the container with id 0d4725e94795749c21e92f3e8c0b283cca9e7ab63a1ee6f7894cd65663d4ef21
Sep 16 18:45:44.680581 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.680551    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0ebe2_8dd0_4bbf_b7a2_3a00a7712069.slice/crio-2866f999c97d73f0e91dd3c84c305b09d3d19c64ac0ee63fdc2dc367fbc4eba4 WatchSource:0}: Error finding container 2866f999c97d73f0e91dd3c84c305b09d3d19c64ac0ee63fdc2dc367fbc4eba4: Status 404 returned error can't find the container with id 2866f999c97d73f0e91dd3c84c305b09d3d19c64ac0ee63fdc2dc367fbc4eba4
Sep 16 18:45:44.681132 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.681109    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7e81bb_1f13_4f36_a344_952e6946dd68.slice/crio-d359284daeb7da51c30e5692b8bf18379c75d99104a7f79af4829ba6ee7fe08b WatchSource:0}: Error finding container d359284daeb7da51c30e5692b8bf18379c75d99104a7f79af4829ba6ee7fe08b: Status 404 returned error can't find the container with id d359284daeb7da51c30e5692b8bf18379c75d99104a7f79af4829ba6ee7fe08b
Sep 16 18:45:44.682953 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.682845    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2554090_4b49_4158_9277_23fd3dbe8ac7.slice/crio-345f9453db1387b0e4d0ac0bc182670fd03585a12d6bafcd5a9969f429492203 WatchSource:0}: Error finding container 345f9453db1387b0e4d0ac0bc182670fd03585a12d6bafcd5a9969f429492203: Status 404 returned error can't find the container with id 345f9453db1387b0e4d0ac0bc182670fd03585a12d6bafcd5a9969f429492203
Sep 16 18:45:44.684084 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.684062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f35ea27_56a7_4718_9e06_4fbfdb490581.slice/crio-bdbd9ad7f5e3dcdde6b24ba6dabf7acbfd7aeab24137f5e664444f85deecacbe WatchSource:0}: Error finding container bdbd9ad7f5e3dcdde6b24ba6dabf7acbfd7aeab24137f5e664444f85deecacbe: Status 404 returned error can't find the container with id bdbd9ad7f5e3dcdde6b24ba6dabf7acbfd7aeab24137f5e664444f85deecacbe
Sep 16 18:45:44.687800 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:45:44.686132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5e4bc2_3bae_4b20_9f56_2ddc8acbe61b.slice/crio-9778c01beeff6025d470421672f0e1f968989960fc021ec67ed4bfc0b9eec544 WatchSource:0}: Error finding container 9778c01beeff6025d470421672f0e1f968989960fc021ec67ed4bfc0b9eec544: Status 404 returned error can't find the container with id 9778c01beeff6025d470421672f0e1f968989960fc021ec67ed4bfc0b9eec544
Sep 16 18:45:45.069192 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.069145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:45.069351 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.069285    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.069351 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.069349    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.069331624 +0000 UTC m=+6.040808851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.170332 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.170248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:45.170501 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.170410    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.170501 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.170476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.170456927 +0000 UTC m=+6.141934162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.271466 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.271311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:45.271655 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.271507    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:45.271655 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.271526    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:45.271655 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.271538    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.271655 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.271595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.271576498 +0000 UTC m=+6.243053727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.544268 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.544144    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:44 +0000 UTC" deadline="2028-05-07 11:59:57.278091894 +0000 UTC"
Sep 16 18:45:45.544268 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.544213    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14369h14m11.73388317s"
Sep 16 18:45:45.599186 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.599154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:45.599362 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.599300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:45.599431 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.599394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:45.599501 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.599475    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:45.601422 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.601395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:45.601559 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:45.601534    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:45.632452 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.632416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerStarted","Data":"d359284daeb7da51c30e5692b8bf18379c75d99104a7f79af4829ba6ee7fe08b"}
Sep 16 18:45:45.655030 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.654987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d979r" event={"ID":"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b","Type":"ContainerStarted","Data":"9778c01beeff6025d470421672f0e1f968989960fc021ec67ed4bfc0b9eec544"}
Sep 16 18:45:45.664916 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.664878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvsmf" event={"ID":"c614891f-8a28-49d4-994f-30fc9308cfaf","Type":"ContainerStarted","Data":"0fe715e7d63b012e4e4882bd2b5001a569ba8d1a1a9326d2d057fe69f23a22cd"}
Sep 16 18:45:45.668590 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.668443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55hqc" event={"ID":"e2554090-4b49-4158-9277-23fd3dbe8ac7","Type":"ContainerStarted","Data":"345f9453db1387b0e4d0ac0bc182670fd03585a12d6bafcd5a9969f429492203"}
Sep 16 18:45:45.679722 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.679694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b" event={"ID":"82abb29f-3294-46b2-b81f-41d42579b761","Type":"ContainerStarted","Data":"0d4725e94795749c21e92f3e8c0b283cca9e7ab63a1ee6f7894cd65663d4ef21"}
Sep 16 18:45:45.687981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.687953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crttb" event={"ID":"9f35ea27-56a7-4718-9e06-4fbfdb490581","Type":"ContainerStarted","Data":"bdbd9ad7f5e3dcdde6b24ba6dabf7acbfd7aeab24137f5e664444f85deecacbe"}
Sep 16 18:45:45.709449 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.709411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"e0f8c50603c151fcd5728cd660c9ad6386cb1e3f847ba55d90c571948aa524d1"}
Sep 16 18:45:45.719659 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.719624    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b3fe0053cfc1761b28dc935cdabb16d" containerID="4c66be4ab8ca05f19a871afa0f8b9dce20132f90fd20dca6f9180e1dd31418ce" exitCode=0
Sep 16 18:45:45.720016 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.719964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal" event={"ID":"4b3fe0053cfc1761b28dc935cdabb16d","Type":"ContainerDied","Data":"4c66be4ab8ca05f19a871afa0f8b9dce20132f90fd20dca6f9180e1dd31418ce"}
Sep 16 18:45:45.741143 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.741043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal" event={"ID":"d7e51c5c4cf47eed84f85b77e7cf2641","Type":"ContainerStarted","Data":"f8c357a169c0466b12b289568e80fcfeef869577ad9a65edf108a6f9fba401ec"}
Sep 16 18:45:45.748845 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.748814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-52748" event={"ID":"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069","Type":"ContainerStarted","Data":"2866f999c97d73f0e91dd3c84c305b09d3d19c64ac0ee63fdc2dc367fbc4eba4"}
Sep 16 18:45:45.786905 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:45.786868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vpqf8" event={"ID":"6750ad10-4915-4413-a651-f22681422251","Type":"ContainerStarted","Data":"967ccad9bca3022233c48dbbd30f74342df2cd6831eff8b94b4be56dd02365fe"}
Sep 16 18:45:46.820466 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:46.820424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal" event={"ID":"4b3fe0053cfc1761b28dc935cdabb16d","Type":"ContainerStarted","Data":"14ac6011df64aea519c7381a895247cc72b276390144f17f657446ab514d58e7"}
Sep 16 18:45:46.864357 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:46.864288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-130.ec2.internal" podStartSLOduration=3.864265058 podStartE2EDuration="3.864265058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.796359019 +0000 UTC m=+4.767836265" watchObservedRunningTime="2026-09-16 18:45:46.864265058 +0000 UTC m=+5.835742313"
Sep 16 18:45:47.085679 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.085583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:47.085998 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.085761    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:47.085998 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.085826    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.085807283 +0000 UTC m=+10.057284516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:47.186557 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.186515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:47.186737 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.186691    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:47.186799 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.186758    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.186739094 +0000 UTC m=+10.158216368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:47.288029 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.287803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:47.288902 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.288381    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:47.288902 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.288415    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:47.288902 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.288428    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:47.288902 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.288496    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.288475433 +0000 UTC m=+10.259952663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:47.599702 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.599010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:47.599702 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.599148    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:47.599702 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.599560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:47.599702 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.599654    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:47.600349 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:47.600163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:47.600349 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:47.600274    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:49.598644 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:49.598609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:49.599137 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:49.598752    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:49.599207 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:49.599169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:49.599291 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:49.599265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:49.599429 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:49.599411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:49.599546 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:49.599525    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:51.125182 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.125136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:51.125678 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.125348    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:51.125678 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.125418    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.125397437 +0000 UTC m=+18.096874686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:51.226549 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.226501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:51.226747 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.226706    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:51.226809 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.226776    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.22675645 +0000 UTC m=+18.198233694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:51.327672 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.327637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:51.327900 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.327794    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:51.327900 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.327811    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:51.327900 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.327822    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:51.327900 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.327869    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.327854853 +0000 UTC m=+18.299332081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:51.599110 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.599017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:51.599110 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.599042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:51.599319 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.599144    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:51.599535 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:51.599511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:51.599662 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.599630    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:51.599796 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:51.599737    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:53.600222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:53.600356    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:53.600725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:53.600839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:53.600904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:53.601022 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:53.600971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:55.597791 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:55.597754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:55.598214 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:55.597761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:55.598214 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:55.597894    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:55.598214 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:55.597760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:55.598214 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:55.597991    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:55.598214 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:55.598018    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:57.597628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:57.597648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:57.597756    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:57.597837    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:57.597900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:57.598125 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:57.598007    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:45:59.186230 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.186194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:59.186704 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.186327    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:59.186704 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.186375    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.186361699 +0000 UTC m=+34.157838939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:59.287163 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.287134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:59.287311 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.287240    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:59.287311 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.287293    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.287275816 +0000 UTC m=+34.258753048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:59.387424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.387389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:59.387580 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.387527    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:59.387580 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.387541    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:59.387580 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.387550    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:59.387690 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.387602    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.387583476 +0000 UTC m=+34.359060704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:59.598706 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.598605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:45:59.598706 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.598641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:45:59.598706 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:45:59.598624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:45:59.598962 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.598728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:45:59.598962 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.598812    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:45:59.598962 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:45:59.598899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:01.598777 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:01.598729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:01.599207 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:01.598839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:01.599207 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:01.598904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:01.599207 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:01.598979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:01.599207 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:01.599006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:01.599207 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:01.599112    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:03.598340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.598049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:03.599055 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.598049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:03.599055 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:03.598429    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:03.599055 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:03.598513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:03.599055 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.598072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:03.599055 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:03.598626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:03.858149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.858112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-52748" event={"ID":"9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069","Type":"ContainerStarted","Data":"cdb0c1c55761e024b2c8ab7938b76ad67d75fcda9f67bf4bba89110dc0b5717e"}
Sep 16 18:46:03.861682 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.861653    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="9e3b9b4af0b771ad04c2ae2eaa08b7337b4e5f6ed78f0a08a96b6b2f0ab95702" exitCode=0
Sep 16 18:46:03.861822 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.861746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"9e3b9b4af0b771ad04c2ae2eaa08b7337b4e5f6ed78f0a08a96b6b2f0ab95702"}
Sep 16 18:46:03.864766 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.864741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d979r" event={"ID":"cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b","Type":"ContainerStarted","Data":"d101e0bcffa6caad54041ea4019ce5ea3727f1f37503ad3f69614091e65400a8"}
Sep 16 18:46:03.866767 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.866744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvsmf" event={"ID":"c614891f-8a28-49d4-994f-30fc9308cfaf","Type":"ContainerStarted","Data":"ad392beeb962d189bd4a5e34ea8ec9abf3b4803a0bc0d27596e052413b613968"}
Sep 16 18:46:03.868434 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.868411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55hqc" event={"ID":"e2554090-4b49-4158-9277-23fd3dbe8ac7","Type":"ContainerStarted","Data":"762d8f58c23a202d324c1af2826ad5ed7a851b5d3ee6ffafe0b7723875980b19"}
Sep 16 18:46:03.871841 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.871439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b" event={"ID":"82abb29f-3294-46b2-b81f-41d42579b761","Type":"ContainerStarted","Data":"aa0dae84ab42312647acb112103adcdc86b3a368012b3055d1c97e1fe08c25aa"}
Sep 16 18:46:03.873337 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.873301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-52748" podStartSLOduration=3.427378546 podStartE2EDuration="21.873290112s" totalImagesPullingTime="18.445911566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.873194499 +0000 UTC m=+22.844671749" watchObservedRunningTime="2026-09-16 18:46:03.873290112 +0000 UTC m=+22.844767361"
Sep 16 18:46:03.873656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.873623    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-130.ec2.internal" podStartSLOduration=20.873613167 podStartE2EDuration="20.873613167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.86519958 +0000 UTC m=+5.836676834" watchObservedRunningTime="2026-09-16 18:46:03.873613167 +0000 UTC m=+22.845090416"
Sep 16 18:46:03.874805 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.874783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crttb" event={"ID":"9f35ea27-56a7-4718-9e06-4fbfdb490581","Type":"ContainerStarted","Data":"431a2d425b968459e68063b2eb0bee01bf4d74f3f54bb77c892f8d6685dc031e"}
Sep 16 18:46:03.882778 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"e6cbcd03101f6b58df93b60dcce1b11ea167c2f51980a5d31f7b2f43d2676708"}
Sep 16 18:46:03.882884 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"89e19ca32336fdd0d205890455cf7d389bd240f88fb5fd0288b0293a2d04c1bc"}
Sep 16 18:46:03.882884 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"e7f11dd7834a568dade4a1d1d38a08ed9fb8f739db0ed8e5f6af4e40e21043b5"}
Sep 16 18:46:03.882987 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"05874c2b70001aee6d8d08fc62e9504c928937e2ddee92d9988538cdc06ab484"}
Sep 16 18:46:03.882987 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"9ad4361289f464ba1dbfd790d040ecdf3abc210883f03ba6cac55879ad2702af"}
Sep 16 18:46:03.882987 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.882950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"319f614264172ff2287fd66d2261ee3cf330728bf6846fc6f88067c30b4735cd"}
Sep 16 18:46:03.966209 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.966164    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lvsmf" podStartSLOduration=3.522618157 podStartE2EDuration="21.966148606s" totalImagesPullingTime="18.443530449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.965931397 +0000 UTC m=+22.937408646" watchObservedRunningTime="2026-09-16 18:46:03.966148606 +0000 UTC m=+22.937625854"
Sep 16 18:46:03.966380 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:03.966253    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d979r" podStartSLOduration=3.526024275 podStartE2EDuration="21.966248545s" totalImagesPullingTime="18.44022427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.940525323 +0000 UTC m=+22.912002571" watchObservedRunningTime="2026-09-16 18:46:03.966248545 +0000 UTC m=+22.937725796"
Sep 16 18:46:04.002901 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.002832    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-crttb" podStartSLOduration=7.482096986 podStartE2EDuration="22.002814615s" totalImagesPullingTime="14.520717629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.002570279 +0000 UTC m=+22.974047539" watchObservedRunningTime="2026-09-16 18:46:04.002814615 +0000 UTC m=+22.974291865"
Sep 16 18:46:04.003079 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.003056    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-55hqc" podStartSLOduration=3.489706964 podStartE2EDuration="22.003049368s" totalImagesPullingTime="18.513342404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.985577924 +0000 UTC m=+22.957055174" watchObservedRunningTime="2026-09-16 18:46:04.003049368 +0000 UTC m=+22.974526616"
Sep 16 18:46:04.360082 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.360048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:46:04.360560 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.360539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:46:04.597204 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.597163    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:46:04.888036 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.888002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b" event={"ID":"82abb29f-3294-46b2-b81f-41d42579b761","Type":"ContainerStarted","Data":"9aa18f9da0eeb8e682f0cff8b56e978c73dc5a24f9096fd9888ddc8e9cf43fee"}
Sep 16 18:46:04.889791 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.889758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vpqf8" event={"ID":"6750ad10-4915-4413-a651-f22681422251","Type":"ContainerStarted","Data":"1af358a9f44d6f4478333bbd01f8836cc3b2834aea1a77898a272d19dfa23a2f"}
Sep 16 18:46:04.890706 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.890686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:46:04.891212 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.891193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crttb"
Sep 16 18:46:04.908244 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:04.908191    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vpqf8" podStartSLOduration=4.444929417 podStartE2EDuration="22.908175613s" totalImagesPullingTime="18.463246196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.907685318 +0000 UTC m=+23.879162567" watchObservedRunningTime="2026-09-16 18:46:04.908175613 +0000 UTC m=+23.879652863"
Sep 16 18:46:05.508776 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.508678    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:46:04.59718563Z","UUID":"5debd76a-2496-4faa-aca0-cd2557a5ca1d","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:46:05.510538 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.510518    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:46:05.510668 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.510556    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:46:05.597820 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.597796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:05.597962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.597831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:05.597962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.597796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:05.598458 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:05.598044    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:05.598458 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:05.598274    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:05.598458 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:05.598387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:05.894360 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:05.894321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b" event={"ID":"82abb29f-3294-46b2-b81f-41d42579b761","Type":"ContainerStarted","Data":"9474d28bc6afa2025e9c45aa097a73a2abfc16e51ccbbc791bf43431a81028e3"}
Sep 16 18:46:06.900578 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:06.900539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"31b2cb508f4e398f765ba267c0e743fc7e7dc9443ca0a49f801bd0be12512f5d"}
Sep 16 18:46:07.597663 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:07.597621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:07.597867 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:07.597723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:07.598156 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:07.598136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:07.598258 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:07.598210    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:07.598318 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:07.598276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:07.598371 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:07.598359    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:08.908801 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.908577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" event={"ID":"a1a862c7-4d03-440d-a483-b0dfb7f21042","Type":"ContainerStarted","Data":"5ecd8ed0c21fff4957a72a829c4a4130124ec832f00aa1d07326eba4d708a1d7"}
Sep 16 18:46:08.909639 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.908875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:08.910630 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.910605    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="f9caa8f744012f8fe4ce9b3fae215808bc4cec8b63066e8cdc8dd2ff176caafa" exitCode=0
Sep 16 18:46:08.910750 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.910634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"f9caa8f744012f8fe4ce9b3fae215808bc4cec8b63066e8cdc8dd2ff176caafa"}
Sep 16 18:46:08.924879 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.924839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:08.953129 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.952969    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m" podStartSLOduration=8.437052843 podStartE2EDuration="26.952951418s" totalImagesPullingTime="18.515898575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:08.951630703 +0000 UTC m=+27.923107952" watchObservedRunningTime="2026-09-16 18:46:08.952951418 +0000 UTC m=+27.924428670"
Sep 16 18:46:08.953731 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:08.953689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g888b" podStartSLOduration=6.482585114 podStartE2EDuration="26.953672357s" totalImagesPullingTime="20.471087243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:05.915732038 +0000 UTC m=+24.887209309" watchObservedRunningTime="2026-09-16 18:46:08.953672357 +0000 UTC m=+27.925149607"
Sep 16 18:46:09.598374 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.598347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:09.598548 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:09.598449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:09.598548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.598349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:09.598548 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:09.598506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:09.598548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.598515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:09.598741 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:09.598651    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:09.913792 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.913587    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:09.914197 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.913874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:09.931022 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:09.931001    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:10.188128 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.188025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsm6d"]
Sep 16 18:46:10.188128 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.188120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:10.188299 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:10.188194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:10.191252 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.191228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpmsl"]
Sep 16 18:46:10.191363 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.191310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:10.191417 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:10.191398    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:10.200845 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.200824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwm6b"]
Sep 16 18:46:10.200940 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.200912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:10.201002 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:10.200982    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:10.918688 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.918650    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="6b9ba4f0e229dedff8361972d4c44de5373a6d42ac473066627e8c5beb76a93b" exitCode=0
Sep 16 18:46:10.919119 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:10.918741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"6b9ba4f0e229dedff8361972d4c44de5373a6d42ac473066627e8c5beb76a93b"}
Sep 16 18:46:11.599130 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:11.599100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:11.599268 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:11.599187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:11.599268 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:11.599235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:11.599344 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:11.599324    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:12.597893 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:12.597865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:12.598264 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:12.597977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:12.925621 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:12.925543    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="e87df074dededc9007f314a3f8f697c8641a10732fa4dd935388b0f2da4b2938" exitCode=0
Sep 16 18:46:12.925621 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:12.925583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"e87df074dededc9007f314a3f8f697c8641a10732fa4dd935388b0f2da4b2938"}
Sep 16 18:46:13.597982 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:13.597743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:13.598432 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:13.597812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:13.598432 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:13.598094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:13.598432 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:13.598206    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:14.598294 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:14.598259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:14.598781 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:14.598411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:15.212280 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:15.212245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:15.212455 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.212393    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:15.212522 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.212465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret podName:96514ca7-0140-4654-9b6c-7c16f636afd6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:47.212443342 +0000 UTC m=+66.183920573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret") pod "global-pull-secret-syncer-mwm6b" (UID: "96514ca7-0140-4654-9b6c-7c16f636afd6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:15.312690 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:15.312663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:15.312844 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.312771    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:15.312844 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.312826    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:47.312810997 +0000 UTC m=+66.284288227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:15.413082 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:15.413038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:15.413268 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.413241    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:15.413268 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.413268    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:15.413373 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.413279    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pdpjp for pod openshift-network-diagnostics/network-check-target-xsm6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:15.413373 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.413345    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp podName:3f558960-469f-40f1-9b7d-2a9208bc2260 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:47.413325492 +0000 UTC m=+66.384802724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pdpjp" (UniqueName: "kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp") pod "network-check-target-xsm6d" (UID: "3f558960-469f-40f1-9b7d-2a9208bc2260") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:15.598099 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:15.598068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:15.598296 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:15.598068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:15.598296 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.598207    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsm6d" podUID="3f558960-469f-40f1-9b7d-2a9208bc2260"
Sep 16 18:46:15.598681 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:15.598289    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwm6b" podUID="96514ca7-0140-4654-9b6c-7c16f636afd6"
Sep 16 18:46:16.597844 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:16.597817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:16.598017 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:16.597969    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zpmsl" podUID="e86c725d-a1ce-429e-8073-83919fcff64b"
Sep 16 18:46:17.303559 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.303478    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-130.ec2.internal" event="NodeReady"
Sep 16 18:46:17.304116 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.303610    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:46:17.356840 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.356814    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2wq27"]
Sep 16 18:46:17.358638 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.358623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.361576 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.361557    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fcv7l\""
Sep 16 18:46:17.361576 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.361568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:46:17.361827 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.361811    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:46:17.366095 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.366076    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2pnvn"]
Sep 16 18:46:17.367970 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.367954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.374718 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.374696    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:46:17.374966 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.374949    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:46:17.375041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.375025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wm755\""
Sep 16 18:46:17.375199 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.375180    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:46:17.377210 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.377190    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2wq27"]
Sep 16 18:46:17.397952 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.397926    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2pnvn"]
Sep 16 18:46:17.529053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.529216 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71efd6a1-843c-4d6c-a064-2e62158db0ef-tmp-dir\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.529216 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71efd6a1-843c-4d6c-a064-2e62158db0ef-config-volume\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.529216 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4befc80-f5e8-4515-b8fb-94796b62d41e-cert\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.529216 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkmz\" (UniqueName: \"kubernetes.io/projected/c4befc80-f5e8-4515-b8fb-94796b62d41e-kube-api-access-cqkmz\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.529363 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.529216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftnv\" (UniqueName: \"kubernetes.io/projected/71efd6a1-843c-4d6c-a064-2e62158db0ef-kube-api-access-sftnv\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.598437 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.598414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:17.598641 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.598618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:17.601755 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.601736    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:17.601977 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.601963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hcvfn\""
Sep 16 18:46:17.602082 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.602066    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:17.602126 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.602099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:17.630150 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.630262 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71efd6a1-843c-4d6c-a064-2e62158db0ef-tmp-dir\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.630322 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:17.630292    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.630371 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71efd6a1-843c-4d6c-a064-2e62158db0ef-config-volume\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.630371 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:17.630364    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.13034225 +0000 UTC m=+37.101819496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.630483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4befc80-f5e8-4515-b8fb-94796b62d41e-cert\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.630483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71efd6a1-843c-4d6c-a064-2e62158db0ef-tmp-dir\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.630483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkmz\" (UniqueName: \"kubernetes.io/projected/c4befc80-f5e8-4515-b8fb-94796b62d41e-kube-api-access-cqkmz\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.630483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftnv\" (UniqueName: \"kubernetes.io/projected/71efd6a1-843c-4d6c-a064-2e62158db0ef-kube-api-access-sftnv\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.630731 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.630705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71efd6a1-843c-4d6c-a064-2e62158db0ef-config-volume\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.634646 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.634623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4befc80-f5e8-4515-b8fb-94796b62d41e-cert\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.647084 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.647062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkmz\" (UniqueName: \"kubernetes.io/projected/c4befc80-f5e8-4515-b8fb-94796b62d41e-kube-api-access-cqkmz\") pod \"ingress-canary-2pnvn\" (UID: \"c4befc80-f5e8-4515-b8fb-94796b62d41e\") " pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.647543 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.647513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftnv\" (UniqueName: \"kubernetes.io/projected/71efd6a1-843c-4d6c-a064-2e62158db0ef-kube-api-access-sftnv\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:17.675027 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.674998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2pnvn"
Sep 16 18:46:17.939863 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.939784    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="eb2ec1035735b0b2d7d5154a74fe0d440041cabce5789533c4e5f03af3ec0532" exitCode=0
Sep 16 18:46:17.939863 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.939829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"eb2ec1035735b0b2d7d5154a74fe0d440041cabce5789533c4e5f03af3ec0532"}
Sep 16 18:46:17.954968 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:17.954941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2pnvn"]
Sep 16 18:46:17.960621 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:46:17.960592    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4befc80_f5e8_4515_b8fb_94796b62d41e.slice/crio-20e21b448e45f71860bb502c3500c10c683c050391d2170c01a9b15e1625f627 WatchSource:0}: Error finding container 20e21b448e45f71860bb502c3500c10c683c050391d2170c01a9b15e1625f627: Status 404 returned error can't find the container with id 20e21b448e45f71860bb502c3500c10c683c050391d2170c01a9b15e1625f627
Sep 16 18:46:18.135494 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.135329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:18.135494 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:18.135480    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.135664 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:18.135548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.135532429 +0000 UTC m=+38.107009655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.598023 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.597989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:18.602188 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.602165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:18.602449 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.602434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzr79\""
Sep 16 18:46:18.943985 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.943907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2pnvn" event={"ID":"c4befc80-f5e8-4515-b8fb-94796b62d41e","Type":"ContainerStarted","Data":"20e21b448e45f71860bb502c3500c10c683c050391d2170c01a9b15e1625f627"}
Sep 16 18:46:18.947761 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.947721    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b7e81bb-1f13-4f36-a344-952e6946dd68" containerID="82db70ff08e5304c42a97e17ed7d700243fc58fa4763f85938e7989c72c96c66" exitCode=0
Sep 16 18:46:18.947925 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:18.947762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerDied","Data":"82db70ff08e5304c42a97e17ed7d700243fc58fa4763f85938e7989c72c96c66"}
Sep 16 18:46:19.145152 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:19.145110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:19.145316 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:19.145258    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:19.145383 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:19.145323    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.145305178 +0000 UTC m=+40.116782406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:19.954759 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:19.954703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" event={"ID":"2b7e81bb-1f13-4f36-a344-952e6946dd68","Type":"ContainerStarted","Data":"b10e540645f25c1b6caf8ba231c5a12351500b099d1c3d48254190effbf5342b"}
Sep 16 18:46:19.982677 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:19.982622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wc6zk" podStartSLOduration=9.16058294 podStartE2EDuration="37.982605943s" totalImagesPullingTime="28.822023003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:19.982204025 +0000 UTC m=+38.953681276" watchObservedRunningTime="2026-09-16 18:46:19.982605943 +0000 UTC m=+38.954083192"
Sep 16 18:46:20.959032 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:20.958999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2pnvn" event={"ID":"c4befc80-f5e8-4515-b8fb-94796b62d41e","Type":"ContainerStarted","Data":"56db6ff62504980338cf06ca65dee8a39fccb537c2b193f638640eedb2443b49"}
Sep 16 18:46:20.979484 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:20.979433    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2pnvn" podStartSLOduration=1.48007987 podStartE2EDuration="3.979414585s" totalImagesPullingTime="2.499334715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:20.97879564 +0000 UTC m=+39.950272890" watchObservedRunningTime="2026-09-16 18:46:20.979414585 +0000 UTC m=+39.950891834"
Sep 16 18:46:21.160210 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:21.159963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:21.160359 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:21.160113    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:21.160359 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:21.160332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:25.160310426 +0000 UTC m=+44.131787657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:25.186354 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:25.186314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:25.186758 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:25.186464    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:25.186758 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:25.186526    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:33.186511385 +0000 UTC m=+52.157988612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:33.240398 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:33.240361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:33.240909 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:33.240505    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:33.240909 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:33.240572    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:46:49.240555669 +0000 UTC m=+68.212032895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:41.937248 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:41.937221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xh5m"
Sep 16 18:46:47.240038 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.239649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:47.243842 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.243818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:47.253978 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.253951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96514ca7-0140-4654-9b6c-7c16f636afd6-original-pull-secret\") pod \"global-pull-secret-syncer-mwm6b\" (UID: \"96514ca7-0140-4654-9b6c-7c16f636afd6\") " pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:47.312648 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.312623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwm6b"
Sep 16 18:46:47.340655 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.340624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:46:47.343894 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.343876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:47.351475 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:47.351449    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:47.351596 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:47.351533    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs podName:e86c725d-a1ce-429e-8073-83919fcff64b nodeName:}" failed. No retries permitted until 2026-09-16 18:47:51.351512739 +0000 UTC m=+130.322989983 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs") pod "network-metrics-daemon-zpmsl" (UID: "e86c725d-a1ce-429e-8073-83919fcff64b") : secret "metrics-daemon-secret" not found
Sep 16 18:46:47.441060 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.441020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:47.444383 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.444363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:47.454666 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.454643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:47.464776 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.464747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpjp\" (UniqueName: \"kubernetes.io/projected/3f558960-469f-40f1-9b7d-2a9208bc2260-kube-api-access-pdpjp\") pod \"network-check-target-xsm6d\" (UID: \"3f558960-469f-40f1-9b7d-2a9208bc2260\") " pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:47.583117 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.583048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwm6b"]
Sep 16 18:46:47.585869 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:46:47.585825    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96514ca7_0140_4654_9b6c_7c16f636afd6.slice/crio-b454332d48071f06458078c487a9f9327381f93ed052bea2a636aacdcea7c912 WatchSource:0}: Error finding container b454332d48071f06458078c487a9f9327381f93ed052bea2a636aacdcea7c912: Status 404 returned error can't find the container with id b454332d48071f06458078c487a9f9327381f93ed052bea2a636aacdcea7c912
Sep 16 18:46:47.610705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.610681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hcvfn\""
Sep 16 18:46:47.618154 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.618142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:47.883553 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:47.883523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsm6d"]
Sep 16 18:46:47.886359 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:46:47.886330    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f558960_469f_40f1_9b7d_2a9208bc2260.slice/crio-0ffec6c18da0c41d5f8e1c3726ae4ff5bbed4b458138a02f9caa3105d232c147 WatchSource:0}: Error finding container 0ffec6c18da0c41d5f8e1c3726ae4ff5bbed4b458138a02f9caa3105d232c147: Status 404 returned error can't find the container with id 0ffec6c18da0c41d5f8e1c3726ae4ff5bbed4b458138a02f9caa3105d232c147
Sep 16 18:46:48.023049 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:48.023014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsm6d" event={"ID":"3f558960-469f-40f1-9b7d-2a9208bc2260","Type":"ContainerStarted","Data":"0ffec6c18da0c41d5f8e1c3726ae4ff5bbed4b458138a02f9caa3105d232c147"}
Sep 16 18:46:48.024751 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:48.024572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mwm6b" event={"ID":"96514ca7-0140-4654-9b6c-7c16f636afd6","Type":"ContainerStarted","Data":"b454332d48071f06458078c487a9f9327381f93ed052bea2a636aacdcea7c912"}
Sep 16 18:46:49.254949 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:49.254911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:46:49.255414 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:49.255092    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:49.255414 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:46:49.255151    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls podName:71efd6a1-843c-4d6c-a064-2e62158db0ef nodeName:}" failed. No retries permitted until 2026-09-16 18:47:21.255133437 +0000 UTC m=+100.226610669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls") pod "dns-default-2wq27" (UID: "71efd6a1-843c-4d6c-a064-2e62158db0ef") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:53.039118 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:53.039080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mwm6b" event={"ID":"96514ca7-0140-4654-9b6c-7c16f636afd6","Type":"ContainerStarted","Data":"26fa4a7197b837bc7f9a44080f78a599a81ad118dddebe564c9d688d9b279378"}
Sep 16 18:46:53.040659 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:53.040633    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsm6d" event={"ID":"3f558960-469f-40f1-9b7d-2a9208bc2260","Type":"ContainerStarted","Data":"be7694fcc8967ae2a264ff3abc031716ef56f662dd40698ad95293170c780705"}
Sep 16 18:46:53.040735 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:53.040687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:46:53.056995 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:53.056954    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mwm6b" podStartSLOduration=66.153146269 podStartE2EDuration="1m11.056938025s" totalImagesPullingTime="4.903791756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.056669635 +0000 UTC m=+72.028146886" watchObservedRunningTime="2026-09-16 18:46:53.056938025 +0000 UTC m=+72.028415275"
Sep 16 18:46:53.084561 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:46:53.084521    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsm6d" podStartSLOduration=66.481462826 podStartE2EDuration="1m11.084507036s" totalImagesPullingTime="4.60304421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.083921182 +0000 UTC m=+72.055398430" watchObservedRunningTime="2026-09-16 18:46:53.084507036 +0000 UTC m=+72.055984284"
Sep 16 18:47:03.589388 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.589277    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-vzbs8"]
Sep 16 18:47:03.593936 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.593910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.596908 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.596890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:47:03.598211 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.598191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:47:03.598305 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.598193    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:47:03.598305 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.598195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qkskw\""
Sep 16 18:47:03.598581 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.598556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:47:03.602991 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.602972    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:47:03.604483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.604460    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vzbs8"]
Sep 16 18:47:03.648436 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-tmp\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.648596 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.648596 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8cjk\" (UniqueName: \"kubernetes.io/projected/37201f6a-b634-4878-80a4-f04088a883db-kube-api-access-g8cjk\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.648596 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-service-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.648596 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37201f6a-b634-4878-80a4-f04088a883db-serving-cert\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.648596 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.648595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-snapshots\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.691801 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.691772    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59898dc88-7qb62"]
Sep 16 18:47:03.693712 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.693681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.697253 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697231    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:47:03.697540 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:47:03.697636 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kwcjg\""
Sep 16 18:47:03.697772 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:47:03.697772 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:47:03.697951 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.697756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:47:03.698273 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.698259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:47:03.715804 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.715776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59898dc88-7qb62"]
Sep 16 18:47:03.749332 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37201f6a-b634-4878-80a4-f04088a883db-serving-cert\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749332 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbd7\" (UniqueName: \"kubernetes.io/projected/ff078225-e337-491b-8f17-6d7464d9b1e4-kube-api-access-flbd7\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-snapshots\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-tmp\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8cjk\" (UniqueName: \"kubernetes.io/projected/37201f6a-b634-4878-80a4-f04088a883db-kube-api-access-g8cjk\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-default-certificate\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-stats-auth\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.749573 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-service-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.749966 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.749600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.750062 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.750006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-tmp\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.750154 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.750133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37201f6a-b634-4878-80a4-f04088a883db-snapshots\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.750344 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.750323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-service-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.750821 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.750805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37201f6a-b634-4878-80a4-f04088a883db-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.751895 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.751874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37201f6a-b634-4878-80a4-f04088a883db-serving-cert\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.760537 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.760517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8cjk\" (UniqueName: \"kubernetes.io/projected/37201f6a-b634-4878-80a4-f04088a883db-kube-api-access-g8cjk\") pod \"insights-operator-5844b856db-vzbs8\" (UID: \"37201f6a-b634-4878-80a4-f04088a883db\") " pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.850953 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.850869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-default-certificate\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.850953 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.850904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.850953 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.850925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-stats-auth\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.850953 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.850954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.851225 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.850977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbd7\" (UniqueName: \"kubernetes.io/projected/ff078225-e337-491b-8f17-6d7464d9b1e4-kube-api-access-flbd7\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.851225 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:03.851009    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:03.851225 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:03.851112    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.351087866 +0000 UTC m=+83.322565093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : secret "router-metrics-certs-default" not found
Sep 16 18:47:03.851225 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:03.851151    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.351120402 +0000 UTC m=+83.322597636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:03.853620 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.853601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-stats-auth\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.853688 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.853666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-default-certificate\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.861037 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.861018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbd7\" (UniqueName: \"kubernetes.io/projected/ff078225-e337-491b-8f17-6d7464d9b1e4-kube-api-access-flbd7\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:03.903043 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.903012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vzbs8"
Sep 16 18:47:03.993339 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.993297    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"]
Sep 16 18:47:03.995182 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:03.995165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.001994 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.001969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:47:04.001994 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.001984    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:47:04.002153 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.001998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:47:04.002153 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.002034    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:47:04.002565 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.002546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ztzsg\""
Sep 16 18:47:04.016741 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.016716    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"]
Sep 16 18:47:04.052950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.052915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4779c1e-4194-4174-9701-191283613252-serving-cert\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.052950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.052949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl8v\" (UniqueName: \"kubernetes.io/projected/b4779c1e-4194-4174-9701-191283613252-kube-api-access-rnl8v\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.053140 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.053011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4779c1e-4194-4174-9701-191283613252-config\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.089186 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.089154    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:04.091016 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.091001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.095574 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.095555    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:47:04.096174 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.096150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:47:04.096281 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.096150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:47:04.096341 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.096297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fdc5l\""
Sep 16 18:47:04.113062 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.112990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:47:04.117648 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.117626    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:04.153896 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.153871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4779c1e-4194-4174-9701-191283613252-config\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.154053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.153922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.153947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.153967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.153985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154053 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4779c1e-4194-4174-9701-191283613252-serving-cert\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.154295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl8v\" (UniqueName: \"kubernetes.io/projected/b4779c1e-4194-4174-9701-191283613252-kube-api-access-rnl8v\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.154295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqtf\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.154540 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.154518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4779c1e-4194-4174-9701-191283613252-config\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.156347 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.156329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4779c1e-4194-4174-9701-191283613252-serving-cert\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.170358 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.170338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl8v\" (UniqueName: \"kubernetes.io/projected/b4779c1e-4194-4174-9701-191283613252-kube-api-access-rnl8v\") pod \"service-ca-operator-cf7c776b7-n2qtr\" (UID: \"b4779c1e-4194-4174-9701-191283613252\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.179132 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.179108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vzbs8"]
Sep 16 18:47:04.181914 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:04.181890    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37201f6a_b634_4878_80a4_f04088a883db.slice/crio-7ff574de7aa0036dac3f1532db059e19501d79eeb1d4c00df343070424c381a6 WatchSource:0}: Error finding container 7ff574de7aa0036dac3f1532db059e19501d79eeb1d4c00df343070424c381a6: Status 404 returned error can't find the container with id 7ff574de7aa0036dac3f1532db059e19501d79eeb1d4c00df343070424c381a6
Sep 16 18:47:04.254959 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.254926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.254959 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.254961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.254997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqtf\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255597 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.255385    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:04.255597 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.255405    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59595b74b6-g2knc: secret "image-registry-tls" not found
Sep 16 18:47:04.255597 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.255597 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.255469    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls podName:3250f042-b8b5-4c5a-89ab-d0c94484a9f9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.75544621 +0000 UTC m=+83.726923437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls") pod "image-registry-59595b74b6-g2knc" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9") : secret "image-registry-tls" not found
Sep 16 18:47:04.255900 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.255620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.256290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.256270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.257607 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.257586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.257722 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.257705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.264915 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.264888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqtf\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.264915 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.264907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.303904 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.303859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"
Sep 16 18:47:04.356434 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.356406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:04.356548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.356463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:04.356612 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.356570    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:04.356664 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.356638    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.356616874 +0000 UTC m=+84.328094123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:04.356664 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.356660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.356649895 +0000 UTC m=+84.328127127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : secret "router-metrics-certs-default" not found
Sep 16 18:47:04.574865 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.574771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr"]
Sep 16 18:47:04.577221 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:04.577192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4779c1e_4194_4174_9701_191283613252.slice/crio-02bfe0140f785eb48c0f72a9727aeafe0fa0d695f7ad782dc2915fbe2a32fe79 WatchSource:0}: Error finding container 02bfe0140f785eb48c0f72a9727aeafe0fa0d695f7ad782dc2915fbe2a32fe79: Status 404 returned error can't find the container with id 02bfe0140f785eb48c0f72a9727aeafe0fa0d695f7ad782dc2915fbe2a32fe79
Sep 16 18:47:04.760433 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:04.760400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:04.760819 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.760503    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:04.760819 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.760515    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59595b74b6-g2knc: secret "image-registry-tls" not found
Sep 16 18:47:04.760819 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:04.760563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls podName:3250f042-b8b5-4c5a-89ab-d0c94484a9f9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.760550312 +0000 UTC m=+84.732027539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls") pod "image-registry-59595b74b6-g2knc" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9") : secret "image-registry-tls" not found
Sep 16 18:47:05.072598 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:05.072553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr" event={"ID":"b4779c1e-4194-4174-9701-191283613252","Type":"ContainerStarted","Data":"02bfe0140f785eb48c0f72a9727aeafe0fa0d695f7ad782dc2915fbe2a32fe79"}
Sep 16 18:47:05.075378 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:05.075346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vzbs8" event={"ID":"37201f6a-b634-4878-80a4-f04088a883db","Type":"ContainerStarted","Data":"7ff574de7aa0036dac3f1532db059e19501d79eeb1d4c00df343070424c381a6"}
Sep 16 18:47:05.365017 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:05.364961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:05.365189 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:05.365038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:05.365189 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.365130    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:05.365189 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.365176    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.365157448 +0000 UTC m=+86.336634680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:05.365337 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.365209    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.365187968 +0000 UTC m=+86.336665213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : secret "router-metrics-certs-default" not found
Sep 16 18:47:05.769496 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:05.769413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:05.769948 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.769590    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:05.769948 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.769615    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59595b74b6-g2knc: secret "image-registry-tls" not found
Sep 16 18:47:05.769948 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:05.769685    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls podName:3250f042-b8b5-4c5a-89ab-d0c94484a9f9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.769664599 +0000 UTC m=+86.741141866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls") pod "image-registry-59595b74b6-g2knc" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9") : secret "image-registry-tls" not found
Sep 16 18:47:07.082673 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.082638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vzbs8" event={"ID":"37201f6a-b634-4878-80a4-f04088a883db","Type":"ContainerStarted","Data":"596b7b1625e5b39124df62621d19d1b45537377f8b8db50429a2c193ce8739d2"}
Sep 16 18:47:07.084080 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.084057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr" event={"ID":"b4779c1e-4194-4174-9701-191283613252","Type":"ContainerStarted","Data":"487f27d40472203592ee73658cfb35e24e6bb5e7dee5d648f296f8c258bcf26e"}
Sep 16 18:47:07.101705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.101663    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-vzbs8" podStartSLOduration=1.924783828 podStartE2EDuration="4.101647998s" totalImagesPullingTime="2.17686417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:07.100924053 +0000 UTC m=+86.072401301" watchObservedRunningTime="2026-09-16 18:47:07.101647998 +0000 UTC m=+86.073125296"
Sep 16 18:47:07.120021 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.119972    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr" podStartSLOduration=2.315746282 podStartE2EDuration="4.119953068s" totalImagesPullingTime="1.804206786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:07.118635332 +0000 UTC m=+86.090112581" watchObservedRunningTime="2026-09-16 18:47:07.119953068 +0000 UTC m=+86.091430318"
Sep 16 18:47:07.335367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.335336    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mfpwz"]
Sep 16 18:47:07.338040 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.338021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.341792 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.341769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:07.342074 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.342058    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:07.342149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.342093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h44qh\""
Sep 16 18:47:07.364265 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.364241    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mfpwz"]
Sep 16 18:47:07.383018 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.382992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:07.383143 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p665l\" (UniqueName: \"kubernetes.io/projected/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-api-access-p665l\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.383143 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-crio-socket\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.383143 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-data-volume\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.383143 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.383137    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:07.383343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.383343 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.383195    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:11.383180741 +0000 UTC m=+90.354657968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : secret "router-metrics-certs-default" not found
Sep 16 18:47:07.383343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.383343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.383252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:07.383343 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.383337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:11.383329865 +0000 UTC m=+90.354807092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:07.483748 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p665l\" (UniqueName: \"kubernetes.io/projected/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-api-access-p665l\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483748 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-crio-socket\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-data-volume\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.483880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-crio-socket\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.483924    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.483941    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.483974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.983960041 +0000 UTC m=+86.955437267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.483999 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.483998    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.983980103 +0000 UTC m=+86.955457343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.484316 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.484129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-data-volume\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.498970 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.498945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p665l\" (UniqueName: \"kubernetes.io/projected/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-api-access-p665l\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.785296 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.785204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:07.785459 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.785372    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:07.785459 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.785393    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59595b74b6-g2knc: secret "image-registry-tls" not found
Sep 16 18:47:07.785584 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.785461    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls podName:3250f042-b8b5-4c5a-89ab-d0c94484a9f9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:11.785442335 +0000 UTC m=+90.756919566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls") pod "image-registry-59595b74b6-g2knc" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9") : secret "image-registry-tls" not found
Sep 16 18:47:07.987649 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.987611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.987649 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:07.987648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:07.987886 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.987734    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.987886 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.987765    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.987886 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.987788    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:08.987772186 +0000 UTC m=+87.959249413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:07.987886 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:07.987818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:08.987804609 +0000 UTC m=+87.959281835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:08.994985 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:08.994942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:08.994985 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:08.994985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:08.995372 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:08.995052    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:08.995372 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:08.995088    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:08.995372 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:08.995106    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.995091019 +0000 UTC m=+89.966568246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:08.995372 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:08.995149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.995134699 +0000 UTC m=+89.966611926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:09.217591 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.217565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d979r_cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b/dns-node-resolver/0.log"
Sep 16 18:47:09.277735 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.277667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"]
Sep 16 18:47:09.279860 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.279831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"
Sep 16 18:47:09.283095 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.283072    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:47:09.283560 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.283528    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:47:09.284433 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.284414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nnpqq\""
Sep 16 18:47:09.292317 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.292294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"]
Sep 16 18:47:09.397816 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.397784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2gt\" (UniqueName: \"kubernetes.io/projected/722e7455-366c-4769-be97-2bbed814df49-kube-api-access-lm2gt\") pod \"migrator-8b4d697c5-wr57z\" (UID: \"722e7455-366c-4769-be97-2bbed814df49\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"
Sep 16 18:47:09.498594 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.498553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2gt\" (UniqueName: \"kubernetes.io/projected/722e7455-366c-4769-be97-2bbed814df49-kube-api-access-lm2gt\") pod \"migrator-8b4d697c5-wr57z\" (UID: \"722e7455-366c-4769-be97-2bbed814df49\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"
Sep 16 18:47:09.510290 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.510262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2gt\" (UniqueName: \"kubernetes.io/projected/722e7455-366c-4769-be97-2bbed814df49-kube-api-access-lm2gt\") pod \"migrator-8b4d697c5-wr57z\" (UID: \"722e7455-366c-4769-be97-2bbed814df49\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"
Sep 16 18:47:09.588504 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.588473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"
Sep 16 18:47:09.858834 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:09.858757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z"]
Sep 16 18:47:09.861662 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:09.861634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e7455_366c_4769_be97_2bbed814df49.slice/crio-ec4560ee1bf4fe8dddcee4e0b17835ba75f9316e8a0ae68c189acd38d8baf489 WatchSource:0}: Error finding container ec4560ee1bf4fe8dddcee4e0b17835ba75f9316e8a0ae68c189acd38d8baf489: Status 404 returned error can't find the container with id ec4560ee1bf4fe8dddcee4e0b17835ba75f9316e8a0ae68c189acd38d8baf489
Sep 16 18:47:10.006198 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:10.006168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-52748_9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069/node-ca/0.log"
Sep 16 18:47:10.092280 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:10.092249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z" event={"ID":"722e7455-366c-4769-be97-2bbed814df49","Type":"ContainerStarted","Data":"ec4560ee1bf4fe8dddcee4e0b17835ba75f9316e8a0ae68c189acd38d8baf489"}
Sep 16 18:47:10.806704 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:10.806672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2pnvn_c4befc80-f5e8-4515-b8fb-94796b62d41e/serve-healthcheck-canary/0.log"
Sep 16 18:47:11.011791 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:11.011749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:11.012278 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:11.011804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:11.012278 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.011932    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:11.012278 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.011946    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:11.012278 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.012001    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.011985687 +0000 UTC m=+93.983462919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:11.012278 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.012018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.012008471 +0000 UTC m=+93.983485699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:11.415493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:11.415411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:11.415493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:11.415458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:11.415667 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.415583    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:11.415667 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.415593    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:19.415573586 +0000 UTC m=+98.387050812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:11.415667 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.415634    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs podName:ff078225-e337-491b-8f17-6d7464d9b1e4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:19.415611646 +0000 UTC m=+98.387088873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs") pod "router-default-59898dc88-7qb62" (UID: "ff078225-e337-491b-8f17-6d7464d9b1e4") : secret "router-metrics-certs-default" not found
Sep 16 18:47:11.818105 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:11.818005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:11.818262 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.818200    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:47:11.818262 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.818224    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59595b74b6-g2knc: secret "image-registry-tls" not found
Sep 16 18:47:11.818383 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:11.818294    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls podName:3250f042-b8b5-4c5a-89ab-d0c94484a9f9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:19.818274636 +0000 UTC m=+98.789751880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls") pod "image-registry-59595b74b6-g2knc" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9") : secret "image-registry-tls" not found
Sep 16 18:47:12.101008 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:12.100966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z" event={"ID":"722e7455-366c-4769-be97-2bbed814df49","Type":"ContainerStarted","Data":"f83ab517524a1095c80e331067b61b1571d7bcc1fd8c5cd519f2a93dfa17f250"}
Sep 16 18:47:12.101379 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:12.101020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z" event={"ID":"722e7455-366c-4769-be97-2bbed814df49","Type":"ContainerStarted","Data":"292f966d2d86d09ccc7714e02eaf8977acdb6e575cd0a481e6f93d85f3ac96b7"}
Sep 16 18:47:12.121900 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:12.121825    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wr57z" podStartSLOduration=1.8188451890000001 podStartE2EDuration="3.121810986s" totalImagesPullingTime="1.302965797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:12.12012699 +0000 UTC m=+91.091604225" watchObservedRunningTime="2026-09-16 18:47:12.121810986 +0000 UTC m=+91.093288235"
Sep 16 18:47:15.044705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:15.044673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:15.044705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:15.044711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:15.045134 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:15.044797    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:15.045134 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:15.044818    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:15.045134 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:15.044879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:23.044841684 +0000 UTC m=+102.016318917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:15.045134 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:15.044894    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls podName:9980b4f7-4c0e-4a6d-965d-3a92021e24e1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:23.04488819 +0000 UTC m=+102.016365417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfpwz" (UID: "9980b4f7-4c0e-4a6d-965d-3a92021e24e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:19.478552 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.478507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:19.478984 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.478577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:19.479181 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.479161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff078225-e337-491b-8f17-6d7464d9b1e4-service-ca-bundle\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:19.481052 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.481034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff078225-e337-491b-8f17-6d7464d9b1e4-metrics-certs\") pod \"router-default-59898dc88-7qb62\" (UID: \"ff078225-e337-491b-8f17-6d7464d9b1e4\") " pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:19.602105 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.602076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:19.880188 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.880149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59898dc88-7qb62"]
Sep 16 18:47:19.882222 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.882196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:19.882590 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:19.882567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff078225_e337_491b_8f17_6d7464d9b1e4.slice/crio-3cd2b4db6676006244425eeb198ae0363d4043da7402ef672e0bb6b18efe545e WatchSource:0}: Error finding container 3cd2b4db6676006244425eeb198ae0363d4043da7402ef672e0bb6b18efe545e: Status 404 returned error can't find the container with id 3cd2b4db6676006244425eeb198ae0363d4043da7402ef672e0bb6b18efe545e
Sep 16 18:47:19.884777 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.884758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"image-registry-59595b74b6-g2knc\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") " pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:19.999247 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:19.999213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:20.126910 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.126873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59898dc88-7qb62" event={"ID":"ff078225-e337-491b-8f17-6d7464d9b1e4","Type":"ContainerStarted","Data":"9e9f2db27d88a643a7b02e5446e585796586438da17c313a50d70c36d85a1f71"}
Sep 16 18:47:20.127047 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.126923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59898dc88-7qb62" event={"ID":"ff078225-e337-491b-8f17-6d7464d9b1e4","Type":"ContainerStarted","Data":"3cd2b4db6676006244425eeb198ae0363d4043da7402ef672e0bb6b18efe545e"}
Sep 16 18:47:20.148370 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.148321    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59898dc88-7qb62" podStartSLOduration=17.148307274 podStartE2EDuration="17.148307274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:20.147671863 +0000 UTC m=+99.119149113" watchObservedRunningTime="2026-09-16 18:47:20.148307274 +0000 UTC m=+99.119784524"
Sep 16 18:47:20.284616 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.284530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:20.288208 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:20.288181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250f042_b8b5_4c5a_89ab_d0c94484a9f9.slice/crio-d5d1db642ffa89ae923d43c476310a03e6c6f648920e0bca3853d8a820f429fa WatchSource:0}: Error finding container d5d1db642ffa89ae923d43c476310a03e6c6f648920e0bca3853d8a820f429fa: Status 404 returned error can't find the container with id d5d1db642ffa89ae923d43c476310a03e6c6f648920e0bca3853d8a820f429fa
Sep 16 18:47:20.603023 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.602982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:20.605455 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:20.605431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:21.131514 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.131473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" event={"ID":"3250f042-b8b5-4c5a-89ab-d0c94484a9f9","Type":"ContainerStarted","Data":"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"}
Sep 16 18:47:21.131707 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.131554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" event={"ID":"3250f042-b8b5-4c5a-89ab-d0c94484a9f9","Type":"ContainerStarted","Data":"d5d1db642ffa89ae923d43c476310a03e6c6f648920e0bca3853d8a820f429fa"}
Sep 16 18:47:21.131707 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.131697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:21.131939 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.131917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:21.132820 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.132795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59898dc88-7qb62"
Sep 16 18:47:21.176237 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.176191    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" podStartSLOduration=17.176168143 podStartE2EDuration="17.176168143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:21.175432151 +0000 UTC m=+100.146909399" watchObservedRunningTime="2026-09-16 18:47:21.176168143 +0000 UTC m=+100.147645391"
Sep 16 18:47:21.295338 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.295296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:47:21.297667 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.297638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71efd6a1-843c-4d6c-a064-2e62158db0ef-metrics-tls\") pod \"dns-default-2wq27\" (UID: \"71efd6a1-843c-4d6c-a064-2e62158db0ef\") " pod="openshift-dns/dns-default-2wq27"
Sep 16 18:47:21.571136 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.571058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fcv7l\""
Sep 16 18:47:21.578334 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.578314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2wq27"
Sep 16 18:47:21.850146 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:21.849973    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2wq27"]
Sep 16 18:47:21.852573 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:21.852546    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71efd6a1_843c_4d6c_a064_2e62158db0ef.slice/crio-c9f72eea56ce05cae9e173cc27bac8ee29b5e294044d1a1523ffa7e145002932 WatchSource:0}: Error finding container c9f72eea56ce05cae9e173cc27bac8ee29b5e294044d1a1523ffa7e145002932: Status 404 returned error can't find the container with id c9f72eea56ce05cae9e173cc27bac8ee29b5e294044d1a1523ffa7e145002932
Sep 16 18:47:22.135767 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:22.135676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wq27" event={"ID":"71efd6a1-843c-4d6c-a064-2e62158db0ef","Type":"ContainerStarted","Data":"c9f72eea56ce05cae9e173cc27bac8ee29b5e294044d1a1523ffa7e145002932"}
Sep 16 18:47:23.109948 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.109906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:23.109948 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.109952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:23.110496 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.110424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:23.112326 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.112300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9980b4f7-4c0e-4a6d-965d-3a92021e24e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfpwz\" (UID: \"9980b4f7-4c0e-4a6d-965d-3a92021e24e1\") " pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:23.245982 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.245948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mfpwz"
Sep 16 18:47:23.863148 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:23.863124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mfpwz"]
Sep 16 18:47:23.936879 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:23.936832    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9980b4f7_4c0e_4a6d_965d_3a92021e24e1.slice/crio-0ca05a60f9a3270253991edfefaf81bbd3b0dd159fabcead5d3850e1937b23ab WatchSource:0}: Error finding container 0ca05a60f9a3270253991edfefaf81bbd3b0dd159fabcead5d3850e1937b23ab: Status 404 returned error can't find the container with id 0ca05a60f9a3270253991edfefaf81bbd3b0dd159fabcead5d3850e1937b23ab
Sep 16 18:47:24.045660 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:24.045632    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsm6d"
Sep 16 18:47:24.144637 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:24.144503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfpwz" event={"ID":"9980b4f7-4c0e-4a6d-965d-3a92021e24e1","Type":"ContainerStarted","Data":"b28a1d2d69e7e2cc25f822215e3c60afc3b05314e5909e7ade50c2751bfcab42"}
Sep 16 18:47:24.144637 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:24.144579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfpwz" event={"ID":"9980b4f7-4c0e-4a6d-965d-3a92021e24e1","Type":"ContainerStarted","Data":"0ca05a60f9a3270253991edfefaf81bbd3b0dd159fabcead5d3850e1937b23ab"}
Sep 16 18:47:24.148080 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:24.148043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wq27" event={"ID":"71efd6a1-843c-4d6c-a064-2e62158db0ef","Type":"ContainerStarted","Data":"5cff01dd842101e4082e256c4aa4aec2b468ac1b41d30f8cc48d5200ef4d19f5"}
Sep 16 18:47:25.153192 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:25.153157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wq27" event={"ID":"71efd6a1-843c-4d6c-a064-2e62158db0ef","Type":"ContainerStarted","Data":"558be33a754f257846b59b1817cd854110b428faf283cde7125890ec14df4372"}
Sep 16 18:47:25.153636 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:25.153252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2wq27"
Sep 16 18:47:25.155077 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:25.155056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfpwz" event={"ID":"9980b4f7-4c0e-4a6d-965d-3a92021e24e1","Type":"ContainerStarted","Data":"ecb2bcab18fa7e366496cbfaf647e5f3ccdf084492b3d8cffbca96cfba73e9a4"}
Sep 16 18:47:25.178981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:25.178937    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2wq27" podStartSLOduration=66.068195211 podStartE2EDuration="1m8.178921769s" totalImagesPullingTime="2.110726558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:25.177485146 +0000 UTC m=+104.148962395" watchObservedRunningTime="2026-09-16 18:47:25.178921769 +0000 UTC m=+104.150399026"
Sep 16 18:47:27.164022 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:27.163986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfpwz" event={"ID":"9980b4f7-4c0e-4a6d-965d-3a92021e24e1","Type":"ContainerStarted","Data":"8e5043cc72ab4167d6053dec22392f3c38b7c92059e3452aec5bca7e3e0cb58b"}
Sep 16 18:47:27.196952 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:27.196906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mfpwz" podStartSLOduration=17.95986228 podStartE2EDuration="20.196887161s" totalImagesPullingTime="2.237024881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:27.195998399 +0000 UTC m=+106.167475650" watchObservedRunningTime="2026-09-16 18:47:27.196887161 +0000 UTC m=+106.168364410"
Sep 16 18:47:33.975794 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:33.975760    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:35.160498 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:35.160470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2wq27"
Sep 16 18:47:39.100221 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.100164    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"]
Sep 16 18:47:39.102930 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.102914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:39.107469 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.107446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s8lwm\""
Sep 16 18:47:39.108724 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.108704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:47:39.130683 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.130662    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"]
Sep 16 18:47:39.229666 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.229634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a23632f-7447-409a-968f-221a047ddb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vs7xn\" (UID: \"8a23632f-7447-409a-968f-221a047ddb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:39.330497 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.330465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a23632f-7447-409a-968f-221a047ddb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vs7xn\" (UID: \"8a23632f-7447-409a-968f-221a047ddb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:39.332992 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.332969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8a23632f-7447-409a-968f-221a047ddb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vs7xn\" (UID: \"8a23632f-7447-409a-968f-221a047ddb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:39.411361 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.411255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:39.688450 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:39.688372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"]
Sep 16 18:47:39.691427 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:39.691402    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a23632f_7447_409a_968f_221a047ddb56.slice/crio-6eb4abb179f423e30ece0200af116708013e11bccf08b9ae001f3c35f75c5f69 WatchSource:0}: Error finding container 6eb4abb179f423e30ece0200af116708013e11bccf08b9ae001f3c35f75c5f69: Status 404 returned error can't find the container with id 6eb4abb179f423e30ece0200af116708013e11bccf08b9ae001f3c35f75c5f69
Sep 16 18:47:40.207349 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.207296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn" event={"ID":"8a23632f-7447-409a-968f-221a047ddb56","Type":"ContainerStarted","Data":"6eb4abb179f423e30ece0200af116708013e11bccf08b9ae001f3c35f75c5f69"}
Sep 16 18:47:40.297948 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.297915    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:47:40.300275 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.300257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.310658 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.310635    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:47:40.321309 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.321254    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:47:40.325838 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.322650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:47:40.325838 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.322971    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2wzjr\""
Sep 16 18:47:40.325838 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.323268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:47:40.325838 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.323491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:47:40.325838 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.324696    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:47:40.326576 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.326541    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:47:40.327922 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.327900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:47:40.329363 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.329341    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:47:40.439761 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.439727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.439954 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.439819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.439954 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.439881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.439954 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.439907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.439954 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.439941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.440104 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.440030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.440104 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.440065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckw5\" (UniqueName: \"kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541291 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541291 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckw5\" (UniqueName: \"kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541291 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541567 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541567 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541672 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.541724 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.541687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.542047 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.542013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.542380 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.542361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.542456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.542361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.542604 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.542582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.544163 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.544142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.544396 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.544313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.552641 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.552620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckw5\" (UniqueName: \"kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5\") pod \"console-6d884856bb-vvmk6\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") " pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.611036 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.611001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:40.958344 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:40.958320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:47:40.960642 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:40.960622    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554ca19e_2fbd_40af_b477_a35600489941.slice/crio-940ff23ab3455fa1423bc6e0142dfcb331c7433de9e553c6027b51eca36799a2 WatchSource:0}: Error finding container 940ff23ab3455fa1423bc6e0142dfcb331c7433de9e553c6027b51eca36799a2: Status 404 returned error can't find the container with id 940ff23ab3455fa1423bc6e0142dfcb331c7433de9e553c6027b51eca36799a2
Sep 16 18:47:41.211832 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:41.211726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn" event={"ID":"8a23632f-7447-409a-968f-221a047ddb56","Type":"ContainerStarted","Data":"356458e93e3f7d805fc39bd55bae4dcb1089f71be242e53e93b51a40c59b76b8"}
Sep 16 18:47:41.212270 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:41.211901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:41.213016 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:41.212986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d884856bb-vvmk6" event={"ID":"554ca19e-2fbd-40af-b477-a35600489941","Type":"ContainerStarted","Data":"940ff23ab3455fa1423bc6e0142dfcb331c7433de9e553c6027b51eca36799a2"}
Sep 16 18:47:41.216599 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:41.216580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn"
Sep 16 18:47:41.349888 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:41.349813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vs7xn" podStartSLOduration=1.334499307 podStartE2EDuration="2.349796875s" totalImagesPullingTime="1.015297568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.257583772 +0000 UTC m=+120.229061021" watchObservedRunningTime="2026-09-16 18:47:41.349796875 +0000 UTC m=+120.321274124"
Sep 16 18:47:42.248889 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.248827    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"]
Sep 16 18:47:42.251435 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.251413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.255083 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.255058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:47:42.255528 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.255497    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7hrlv\""
Sep 16 18:47:42.255879 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.255841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:47:42.256649 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.256624    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:47:42.256735 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.256651    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:47:42.257311 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.257293    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:47:42.265076 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.265052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"]
Sep 16 18:47:42.357183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.357148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcnx\" (UniqueName: \"kubernetes.io/projected/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-kube-api-access-sdcnx\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.357349 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.357198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.357349 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.357233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.357472 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.357377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.458162 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.458128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.458340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.458214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.458340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.458258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcnx\" (UniqueName: \"kubernetes.io/projected/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-kube-api-access-sdcnx\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.458340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.458293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.458502 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:42.458386    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 18:47:42.458502 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:42.458457    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls podName:dffafd27-7d51-4189-83eb-c08dd7fcbb6a nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.958437138 +0000 UTC m=+121.929914382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-8tpfz" (UID: "dffafd27-7d51-4189-83eb-c08dd7fcbb6a") : secret "prometheus-operator-tls" not found
Sep 16 18:47:42.459067 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.459042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.460947 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.460920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.470810 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.470783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcnx\" (UniqueName: \"kubernetes.io/projected/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-kube-api-access-sdcnx\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.963079 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.963040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:42.965829 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:42.965800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dffafd27-7d51-4189-83eb-c08dd7fcbb6a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tpfz\" (UID: \"dffafd27-7d51-4189-83eb-c08dd7fcbb6a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:43.163949 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:43.163909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"
Sep 16 18:47:43.930285 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:43.930204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz"]
Sep 16 18:47:43.933116 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:43.933088    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffafd27_7d51_4189_83eb_c08dd7fcbb6a.slice/crio-ecf209c722f02bc4a88d38cc8e143fa05256d88ceb1a00254248803f9a502618 WatchSource:0}: Error finding container ecf209c722f02bc4a88d38cc8e143fa05256d88ceb1a00254248803f9a502618: Status 404 returned error can't find the container with id ecf209c722f02bc4a88d38cc8e143fa05256d88ceb1a00254248803f9a502618
Sep 16 18:47:43.982103 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:43.982082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:44.225104 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:44.225019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz" event={"ID":"dffafd27-7d51-4189-83eb-c08dd7fcbb6a","Type":"ContainerStarted","Data":"ecf209c722f02bc4a88d38cc8e143fa05256d88ceb1a00254248803f9a502618"}
Sep 16 18:47:44.226569 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:44.226540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d884856bb-vvmk6" event={"ID":"554ca19e-2fbd-40af-b477-a35600489941","Type":"ContainerStarted","Data":"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"}
Sep 16 18:47:44.257198 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:44.257151    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d884856bb-vvmk6" podStartSLOduration=1.544164904 podStartE2EDuration="4.257134951s" totalImagesPullingTime="2.712970047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:44.255869116 +0000 UTC m=+123.227346358" watchObservedRunningTime="2026-09-16 18:47:44.257134951 +0000 UTC m=+123.228612199"
Sep 16 18:47:45.231728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:45.231689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz" event={"ID":"dffafd27-7d51-4189-83eb-c08dd7fcbb6a","Type":"ContainerStarted","Data":"6aa4c8330c7c2309fce23beb018450441ab6a6afac7e6c94e302cc08c5b86304"}
Sep 16 18:47:45.232199 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:45.231751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz" event={"ID":"dffafd27-7d51-4189-83eb-c08dd7fcbb6a","Type":"ContainerStarted","Data":"15d3e14c21546778a77a6647811581f6944f10ce41adf9058374a6ed4072581b"}
Sep 16 18:47:47.913507 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.913444    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tpfz" podStartSLOduration=4.75994453 podStartE2EDuration="5.913427209s" totalImagesPullingTime="1.153482679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:45.315497977 +0000 UTC m=+124.286975225" watchObservedRunningTime="2026-09-16 18:47:47.913427209 +0000 UTC m=+126.884904457"
Sep 16 18:47:47.914073 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.914053    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"]
Sep 16 18:47:47.916418 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.916397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:47.919303 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.919261    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:47:47.919478 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.919460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:47:47.920878 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.920841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:47:47.923115 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.923097    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2fkmd\""
Sep 16 18:47:47.975716 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.975686    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lwrg6"]
Sep 16 18:47:47.978140 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.978121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"]
Sep 16 18:47:47.978228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.978214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:47.983038 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.983006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:47:47.983201 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.983177    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:47:47.983296 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.983181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:47:47.983296 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:47.983238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7p4vc\""
Sep 16 18:47:48.003632 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-root\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.003728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7955p\" (UniqueName: \"kubernetes.io/projected/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-kube-api-access-7955p\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4ww\" (UniqueName: \"kubernetes.io/projected/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-api-access-wj4ww\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.003826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-sys\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.003943 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-metrics-client-ca\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003943 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.003943 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/915c4408-d3bd-431c-8d23-f522f45d8f46-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.004041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.004041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.003985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.004041 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.004005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.004132 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.004059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-textfile\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.004132 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.004096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-wtmp\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104628 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-root\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104782 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.104782 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-root\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104782 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:48.104721    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:47:48.104782 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104782 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:48.104770    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls podName:915c4408-d3bd-431c-8d23-f522f45d8f46 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.604754597 +0000 UTC m=+127.576231837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-z57gz" (UID: "915c4408-d3bd-431c-8d23-f522f45d8f46") : secret "kube-state-metrics-tls" not found
Sep 16 18:47:48.104993 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7955p\" (UniqueName: \"kubernetes.io/projected/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-kube-api-access-7955p\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104993 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4ww\" (UniqueName: \"kubernetes.io/projected/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-api-access-wj4ww\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.104993 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-sys\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.104993 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.104979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-metrics-client-ca\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-sys\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/915c4408-d3bd-431c-8d23-f522f45d8f46-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.105243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.105529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-textfile\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-wtmp\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-wtmp\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/915c4408-d3bd-431c-8d23-f522f45d8f46-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.105794 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:48.105573    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:47:48.105794 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:48.105622    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls podName:0856dc1a-4dba-46a3-9cfb-858b3e515f5e nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.605605085 +0000 UTC m=+127.577082332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls") pod "node-exporter-lwrg6" (UID: "0856dc1a-4dba-46a3-9cfb-858b3e515f5e") : secret "node-exporter-tls" not found
Sep 16 18:47:48.105794 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105794 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-metrics-client-ca\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.105794 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-textfile\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.106032 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.106032 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.105936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.107695 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.107675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.107781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.107732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.127304 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.127277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7955p\" (UniqueName: \"kubernetes.io/projected/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-kube-api-access-7955p\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.128607 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.128586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4ww\" (UniqueName: \"kubernetes.io/projected/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-api-access-wj4ww\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.608367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.608333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.608522 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.608390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.610805 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.610774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0856dc1a-4dba-46a3-9cfb-858b3e515f5e-node-exporter-tls\") pod \"node-exporter-lwrg6\" (UID: \"0856dc1a-4dba-46a3-9cfb-858b3e515f5e\") " pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.610976 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.610957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/915c4408-d3bd-431c-8d23-f522f45d8f46-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-z57gz\" (UID: \"915c4408-d3bd-431c-8d23-f522f45d8f46\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.825241 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.825202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"
Sep 16 18:47:48.890174 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:48.889656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwrg6"
Sep 16 18:47:48.904018 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:48.903992    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0856dc1a_4dba_46a3_9cfb_858b3e515f5e.slice/crio-2b7cb07036c4437aa89c64e2c861a6fd94e292c9a27f3b6502b6a695c5128a60 WatchSource:0}: Error finding container 2b7cb07036c4437aa89c64e2c861a6fd94e292c9a27f3b6502b6a695c5128a60: Status 404 returned error can't find the container with id 2b7cb07036c4437aa89c64e2c861a6fd94e292c9a27f3b6502b6a695c5128a60
Sep 16 18:47:49.114589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:49.114437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz"]
Sep 16 18:47:49.117763 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:49.117736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915c4408_d3bd_431c_8d23_f522f45d8f46.slice/crio-4755b6204844279946d594a958cdff95943752ace27c957d1444cd50563b22b2 WatchSource:0}: Error finding container 4755b6204844279946d594a958cdff95943752ace27c957d1444cd50563b22b2: Status 404 returned error can't find the container with id 4755b6204844279946d594a958cdff95943752ace27c957d1444cd50563b22b2
Sep 16 18:47:49.251009 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:49.250976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz" event={"ID":"915c4408-d3bd-431c-8d23-f522f45d8f46","Type":"ContainerStarted","Data":"4755b6204844279946d594a958cdff95943752ace27c957d1444cd50563b22b2"}
Sep 16 18:47:49.252327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:49.252291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwrg6" event={"ID":"0856dc1a-4dba-46a3-9cfb-858b3e515f5e","Type":"ContainerStarted","Data":"2b7cb07036c4437aa89c64e2c861a6fd94e292c9a27f3b6502b6a695c5128a60"}
Sep 16 18:47:50.258196 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.258086    2578 generic.go:356] "Generic (PLEG): container finished" podID="0856dc1a-4dba-46a3-9cfb-858b3e515f5e" containerID="0affcfd257517b8c2a0aa9407d1311ca39e009e71246f844931a729f4da3220f" exitCode=0
Sep 16 18:47:50.258633 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.258249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwrg6" event={"ID":"0856dc1a-4dba-46a3-9cfb-858b3e515f5e","Type":"ContainerDied","Data":"0affcfd257517b8c2a0aa9407d1311ca39e009e71246f844931a729f4da3220f"}
Sep 16 18:47:50.260189 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.260167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz" event={"ID":"915c4408-d3bd-431c-8d23-f522f45d8f46","Type":"ContainerStarted","Data":"52de19b6223d2da73dd30e7e1f760cacf09fbb45e029beb86a88a999f1aad5e4"}
Sep 16 18:47:50.612329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.612298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:50.612491 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.612350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:50.616876 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.616840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:50.722121 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.722090    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"]
Sep 16 18:47:50.724932 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.724912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.727752 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.727734    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lr9b6\""
Sep 16 18:47:50.728595 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.728578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:47:50.729732 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.729706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:47:50.729837 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.729819    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:47:50.729939 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.729839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e307j97slgbn0\""
Sep 16 18:47:50.729939 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.729917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:47:50.730040 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.729999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:47:50.752055 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.752027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"]
Sep 16 18:47:50.825268 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825423 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825423 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825423 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6569c2-e6d2-4456-a427-0926f57f11f3-metrics-client-ca\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825569 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825569 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825569 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7srv\" (UniqueName: \"kubernetes.io/projected/6a6569c2-e6d2-4456-a427-0926f57f11f3-kube-api-access-p7srv\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.825704 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.825603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-grpc-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926590 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926590 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926590 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7srv\" (UniqueName: \"kubernetes.io/projected/6a6569c2-e6d2-4456-a427-0926f57f11f3-kube-api-access-p7srv\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-grpc-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.926839 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.926830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6569c2-e6d2-4456-a427-0926f57f11f3-metrics-client-ca\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.927800 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.927773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6569c2-e6d2-4456-a427-0926f57f11f3-metrics-client-ca\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.929239 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.929454 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.929769 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.929822 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.929936 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.930002 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.929982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a6569c2-e6d2-4456-a427-0926f57f11f3-secret-grpc-tls\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:50.936065 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:50.936050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7srv\" (UniqueName: \"kubernetes.io/projected/6a6569c2-e6d2-4456-a427-0926f57f11f3-kube-api-access-p7srv\") pod \"thanos-querier-69454d8c7b-wzlzp\" (UID: \"6a6569c2-e6d2-4456-a427-0926f57f11f3\") " pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:51.034425 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.034391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:51.266133 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.266049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz" event={"ID":"915c4408-d3bd-431c-8d23-f522f45d8f46","Type":"ContainerStarted","Data":"ddd8380ab9ba22cc814cdec08ed072f6957f8058b896266072837da2093216d2"}
Sep 16 18:47:51.266133 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.266099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz" event={"ID":"915c4408-d3bd-431c-8d23-f522f45d8f46","Type":"ContainerStarted","Data":"de806318c3594fdf79324294a70485616eb7d0bcd97239b89f8ad5bd3558bfdf"}
Sep 16 18:47:51.268239 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.268201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwrg6" event={"ID":"0856dc1a-4dba-46a3-9cfb-858b3e515f5e","Type":"ContainerStarted","Data":"4c4a42ae2804a2f44d087445211ccdb096a4c48fcb64bc38ef6f9605dea4b902"}
Sep 16 18:47:51.268371 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.268254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwrg6" event={"ID":"0856dc1a-4dba-46a3-9cfb-858b3e515f5e","Type":"ContainerStarted","Data":"b14e283a28ffabfb6cf4c7b99f44d1e8c9124728fb7cb07a419fe16e1cccc08c"}
Sep 16 18:47:51.272392 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.272372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:47:51.320804 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.320760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-z57gz" podStartSLOduration=3.271265627 podStartE2EDuration="4.320743922s" totalImagesPullingTime="1.049478295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.29408468 +0000 UTC m=+130.265561940" watchObservedRunningTime="2026-09-16 18:47:51.320743922 +0000 UTC m=+130.292221171"
Sep 16 18:47:51.321225 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.321208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"]
Sep 16 18:47:51.323124 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:51.323097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6569c2_e6d2_4456_a427_0926f57f11f3.slice/crio-84efcea1c7f3392572d79450537ac0a419f4d7729fa1df137053077cda1c088d WatchSource:0}: Error finding container 84efcea1c7f3392572d79450537ac0a419f4d7729fa1df137053077cda1c088d: Status 404 returned error can't find the container with id 84efcea1c7f3392572d79450537ac0a419f4d7729fa1df137053077cda1c088d
Sep 16 18:47:51.427027 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.426970    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lwrg6" podStartSLOduration=3.503376031 podStartE2EDuration="4.426952028s" totalImagesPullingTime="844.710871ms" totalInitContainerRuntime="78.865126ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.425742715 +0000 UTC m=+130.397219963" watchObservedRunningTime="2026-09-16 18:47:51.426952028 +0000 UTC m=+130.398429276"
Sep 16 18:47:51.432222 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.432189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:47:51.434734 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.434709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e86c725d-a1ce-429e-8073-83919fcff64b-metrics-certs\") pod \"network-metrics-daemon-zpmsl\" (UID: \"e86c725d-a1ce-429e-8073-83919fcff64b\") " pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:47:51.610498 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.610457    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzr79\""
Sep 16 18:47:51.618075 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.618055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zpmsl"
Sep 16 18:47:51.894828 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:51.894742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zpmsl"]
Sep 16 18:47:51.897932 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:51.897904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86c725d_a1ce_429e_8073_83919fcff64b.slice/crio-ccf1d83818ed0ce1ef1ad2b3c86ae2ad6c7969817795648694d3e3652f1d41ed WatchSource:0}: Error finding container ccf1d83818ed0ce1ef1ad2b3c86ae2ad6c7969817795648694d3e3652f1d41ed: Status 404 returned error can't find the container with id ccf1d83818ed0ce1ef1ad2b3c86ae2ad6c7969817795648694d3e3652f1d41ed
Sep 16 18:47:52.279277 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:52.279136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpmsl" event={"ID":"e86c725d-a1ce-429e-8073-83919fcff64b","Type":"ContainerStarted","Data":"ccf1d83818ed0ce1ef1ad2b3c86ae2ad6c7969817795648694d3e3652f1d41ed"}
Sep 16 18:47:52.281381 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:52.281270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"84efcea1c7f3392572d79450537ac0a419f4d7729fa1df137053077cda1c088d"}
Sep 16 18:47:54.085698 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.085661    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:54.101733 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.101701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.106304 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.106276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2f22x\""
Sep 16 18:47:54.106304 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.106293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:47:54.108308 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.107157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:47:54.108308 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.107226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:47:54.109914 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.109652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:47:54.109914 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.109676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:47:54.109914 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.109679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:47:54.110120 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.109998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:47:54.110173 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.110123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:47:54.110229 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.110174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:47:54.112574 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.112553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fhmljpikbjct0\""
Sep 16 18:47:54.112778 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.112763    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:47:54.113346 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.112966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:47:54.113346 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.112765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:47:54.116211 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.116180    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:47:54.117506 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.117480    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:47:54.122121 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.122096    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:47:54.138877 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.138815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:54.157044 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157228 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglx6\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-kube-api-access-hglx6\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.157950 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.157749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.258923 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.258888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259076 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.258936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259253 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259338 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259386 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259386 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259487 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.259717 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.260101 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.260101 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglx6\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-kube-api-access-hglx6\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.260101 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.259824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.260299 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.260174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.260491 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.260460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.260969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.261933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.263405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.263484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.263833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.264233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d698d7e-0ecc-48b3-963b-08bb230d3124-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.264278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.264749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.264344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-config\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.265334 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.265288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d698d7e-0ecc-48b3-963b-08bb230d3124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.265994 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.265950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.266520 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.266489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.266659 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.266521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.266973 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.266952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.267295 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.267281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.267650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.267604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.269052 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.269032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d698d7e-0ecc-48b3-963b-08bb230d3124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.281318 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.281292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglx6\" (UniqueName: \"kubernetes.io/projected/7d698d7e-0ecc-48b3-963b-08bb230d3124-kube-api-access-hglx6\") pod \"prometheus-k8s-0\" (UID: \"7d698d7e-0ecc-48b3-963b-08bb230d3124\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.296714 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.296667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpmsl" event={"ID":"e86c725d-a1ce-429e-8073-83919fcff64b","Type":"ContainerStarted","Data":"f1285b5b3a8b9f9954d72f2f296450cc28ba4919da68669c6a7bd6d90788603b"}
Sep 16 18:47:54.296872 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.296744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zpmsl" event={"ID":"e86c725d-a1ce-429e-8073-83919fcff64b","Type":"ContainerStarted","Data":"401540af576d2fae40888c1e6e8e45f04e7757554c57121ca921e685e056c30d"}
Sep 16 18:47:54.299331 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.299303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"e2f1fc3e01ac8ea1271c4479b3e576885e32ffc5dd963401944edd5f03fe3594"}
Sep 16 18:47:54.299473 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.299420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"24dfdef185b565f69a1aedf2994b74be0f33a4e977da33122da4ef88b75484eb"}
Sep 16 18:47:54.299581 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.299472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"a02dd10318191afdcf1a2528e15573258c69df33cd9877394f95776e93306239"}
Sep 16 18:47:54.339407 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.339293    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zpmsl" podStartSLOduration=130.936429435 podStartE2EDuration="2m12.339269325s" totalImagesPullingTime="1.40283989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:54.338677205 +0000 UTC m=+133.310154458" watchObservedRunningTime="2026-09-16 18:47:54.339269325 +0000 UTC m=+133.310746577"
Sep 16 18:47:54.416498 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.416473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.706393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:54.706320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:54.710966 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:47:54.710941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d698d7e_0ecc_48b3_963b_08bb230d3124.slice/crio-bb530e698558ca30106f868e068b774aec5865fc6b4896631889e0aa4cf9109a WatchSource:0}: Error finding container bb530e698558ca30106f868e068b774aec5865fc6b4896631889e0aa4cf9109a: Status 404 returned error can't find the container with id bb530e698558ca30106f868e068b774aec5865fc6b4896631889e0aa4cf9109a
Sep 16 18:47:55.309315 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.309277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"baa9d21efb51f69892fcf141c8c6e495523e349395828c14abd024d5730ae2c5"}
Sep 16 18:47:55.310066 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.310041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"d816eedfe6830b74a5e7fcc1d596472c149d1e5142610f383605901ad6145996"}
Sep 16 18:47:55.310225 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.310208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" event={"ID":"6a6569c2-e6d2-4456-a427-0926f57f11f3","Type":"ContainerStarted","Data":"055ddba7cd8d6fbea22fd3652954583c64357d6708b3d57ffadb5c345fb08b50"}
Sep 16 18:47:55.310384 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.310371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:47:55.314152 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.314127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"bb530e698558ca30106f868e068b774aec5865fc6b4896631889e0aa4cf9109a"}
Sep 16 18:47:55.359216 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:55.359161    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp" podStartSLOduration=2.506487266 podStartE2EDuration="5.359145549s" totalImagesPullingTime="2.852658283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.358888419 +0000 UTC m=+134.330365668" watchObservedRunningTime="2026-09-16 18:47:55.359145549 +0000 UTC m=+134.330622799"
Sep 16 18:47:56.319840 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:56.319796    2578 generic.go:356] "Generic (PLEG): container finished" podID="7d698d7e-0ecc-48b3-963b-08bb230d3124" containerID="aeccea530ece74ee4a794a859bddec1369839e2b6d090bb750420d26018c3236" exitCode=0
Sep 16 18:47:56.320363 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:56.319885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerDied","Data":"aeccea530ece74ee4a794a859bddec1369839e2b6d090bb750420d26018c3236"}
Sep 16 18:47:58.994872 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:58.994805    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" podUID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" containerName="registry" containerID="cri-o://73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8" gracePeriod=30
Sep 16 18:47:59.253252 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.253223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305636    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.306393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.305762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smqtf\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf\") pod \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\" (UID: \"3250f042-b8b5-4c5a-89ab-d0c94484a9f9\") "
Sep 16 18:47:59.309081 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.308834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:59.309537 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.309501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:59.310539 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.310515    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:59.312773 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.312733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:59.313126 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.313054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf" (OuterVolumeSpecName: "kube-api-access-smqtf") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "kube-api-access-smqtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:59.313308 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.313268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:59.315738 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.315613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:59.324117 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.324087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3250f042-b8b5-4c5a-89ab-d0c94484a9f9" (UID: "3250f042-b8b5-4c5a-89ab-d0c94484a9f9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:59.336127 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.336096    2578 generic.go:356] "Generic (PLEG): container finished" podID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" containerID="73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8" exitCode=0
Sep 16 18:47:59.336396 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.336299    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59595b74b6-g2knc"
Sep 16 18:47:59.337081 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.336561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" event={"ID":"3250f042-b8b5-4c5a-89ab-d0c94484a9f9","Type":"ContainerDied","Data":"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"}
Sep 16 18:47:59.337081 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.336648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59595b74b6-g2knc" event={"ID":"3250f042-b8b5-4c5a-89ab-d0c94484a9f9","Type":"ContainerDied","Data":"d5d1db642ffa89ae923d43c476310a03e6c6f648920e0bca3853d8a820f429fa"}
Sep 16 18:47:59.337081 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.336672    2578 scope.go:117] "RemoveContainer" containerID="73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"
Sep 16 18:47:59.340870 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.340610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"01322a81fd555eb8b47a2e896b8150f9159df756c6e10cfa4e0df606543046f5"}
Sep 16 18:47:59.340870 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.340704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"5b6c6965dcb693f021b82b54874b63054d3db56566d0ad118b6bd59588f64d87"}
Sep 16 18:47:59.350209 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.350185    2578 scope.go:117] "RemoveContainer" containerID="73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"
Sep 16 18:47:59.350540 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:47:59.350511    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8\": container with ID starting with 73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8 not found: ID does not exist" containerID="73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"
Sep 16 18:47:59.350636 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.350553    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8"} err="failed to get container status \"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8\": rpc error: code = NotFound desc = could not find container \"73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8\": container with ID starting with 73c1a087707e4dbef4d3496ffa2b1884640c7a045c15e45572aa6daf51cc1dd8 not found: ID does not exist"
Sep 16 18:47:59.374095 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.374067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:59.382608 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.382420    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59595b74b6-g2knc"]
Sep 16 18:47:59.406599 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.406570    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-certificates\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.406768 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.406750    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-installation-pull-secrets\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.406879 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.406865    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-image-registry-private-configuration\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.406963 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.406951    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-registry-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.407059 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.407047    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-trusted-ca\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.407150 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.407137    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqtf\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-kube-api-access-smqtf\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.407230 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.407219    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-bound-sa-token\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.407305 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.407295    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3250f042-b8b5-4c5a-89ab-d0c94484a9f9-ca-trust-extracted\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.603571 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:47:59.603545    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" path="/var/lib/kubelet/pods/3250f042-b8b5-4c5a-89ab-d0c94484a9f9/volumes"
Sep 16 18:48:00.349904 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:00.349840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"d0215577297bd56e3a287e0af4fd3b19d0a6e403f1d234724e19a192f2fa6363"}
Sep 16 18:48:00.350345 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:00.349939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"79ddd97939635300e6784a46879da82bd2697c6f9fe0f55ef37f8370001b4076"}
Sep 16 18:48:00.350345 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:00.349985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"a3f8a2d80f3326f510665e68628df38d2b7598da65b69c7b5e74a982778c02f4"}
Sep 16 18:48:00.350345 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:00.350015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d698d7e-0ecc-48b3-963b-08bb230d3124","Type":"ContainerStarted","Data":"737fcac10e771228b634caeb531bb9178b2c5ad41b0c3d654f19bae7636d3dd6"}
Sep 16 18:48:00.399510 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:00.399431    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.537599556 podStartE2EDuration="6.399412858s" totalImagesPullingTime="3.843046024s" totalInitContainerRuntime="18.767278ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:00.396406792 +0000 UTC m=+139.367884041" watchObservedRunningTime="2026-09-16 18:48:00.399412858 +0000 UTC m=+139.370890118"
Sep 16 18:48:02.331308 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:02.331277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69454d8c7b-wzlzp"
Sep 16 18:48:04.417307 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:04.417277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:10.393935 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:10.393903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:48:27.458916 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:27.458883    2578 generic.go:356] "Generic (PLEG): container finished" podID="b4779c1e-4194-4174-9701-191283613252" containerID="487f27d40472203592ee73658cfb35e24e6bb5e7dee5d648f296f8c258bcf26e" exitCode=0
Sep 16 18:48:27.459340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:27.458965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr" event={"ID":"b4779c1e-4194-4174-9701-191283613252","Type":"ContainerDied","Data":"487f27d40472203592ee73658cfb35e24e6bb5e7dee5d648f296f8c258bcf26e"}
Sep 16 18:48:27.459340 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:27.459272    2578 scope.go:117] "RemoveContainer" containerID="487f27d40472203592ee73658cfb35e24e6bb5e7dee5d648f296f8c258bcf26e"
Sep 16 18:48:28.465111 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:28.465073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-n2qtr" event={"ID":"b4779c1e-4194-4174-9701-191283613252","Type":"ContainerStarted","Data":"527f5f5cf4ecf83d30304e2f0c054732096f91ad420bfed63f7145fba3243fa5"}
Sep 16 18:48:35.413604 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.413500    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d884856bb-vvmk6" podUID="554ca19e-2fbd-40af-b477-a35600489941" containerName="console" containerID="cri-o://d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20" gracePeriod=15
Sep 16 18:48:35.656165 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.656143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:48:35.812483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812798 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812798 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812532    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckw5\" (UniqueName: \"kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812798 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle\") pod \"554ca19e-2fbd-40af-b477-a35600489941\" (UID: \"554ca19e-2fbd-40af-b477-a35600489941\") "
Sep 16 18:48:35.812988 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config" (OuterVolumeSpecName: "console-config") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:35.812988 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.812882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:35.813125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.813098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:35.813125 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.813108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca" (OuterVolumeSpecName: "service-ca") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:35.815045 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.815016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5" (OuterVolumeSpecName: "kube-api-access-hckw5") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "kube-api-access-hckw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:35.815209 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.815190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:35.815269 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.815214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "554ca19e-2fbd-40af-b477-a35600489941" (UID: "554ca19e-2fbd-40af-b477-a35600489941"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:35.913424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913388    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-trusted-ca-bundle\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913415    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-console-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913428    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-oauth-serving-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913439    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-serving-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913451    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554ca19e-2fbd-40af-b477-a35600489941-console-oauth-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913464    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554ca19e-2fbd-40af-b477-a35600489941-service-ca\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:35.913650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:35.913475    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckw5\" (UniqueName: \"kubernetes.io/projected/554ca19e-2fbd-40af-b477-a35600489941-kube-api-access-hckw5\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:48:36.495334 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.495294    2578 generic.go:356] "Generic (PLEG): container finished" podID="554ca19e-2fbd-40af-b477-a35600489941" containerID="d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20" exitCode=0
Sep 16 18:48:36.495334 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.495334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d884856bb-vvmk6"
Sep 16 18:48:36.495826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.495384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d884856bb-vvmk6" event={"ID":"554ca19e-2fbd-40af-b477-a35600489941","Type":"ContainerDied","Data":"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"}
Sep 16 18:48:36.495826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.495431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d884856bb-vvmk6" event={"ID":"554ca19e-2fbd-40af-b477-a35600489941","Type":"ContainerDied","Data":"940ff23ab3455fa1423bc6e0142dfcb331c7433de9e553c6027b51eca36799a2"}
Sep 16 18:48:36.495826 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.495453    2578 scope.go:117] "RemoveContainer" containerID="d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"
Sep 16 18:48:36.504483 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.504469    2578 scope.go:117] "RemoveContainer" containerID="d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"
Sep 16 18:48:36.504746 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:48:36.504728    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20\": container with ID starting with d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20 not found: ID does not exist" containerID="d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"
Sep 16 18:48:36.504791 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.504756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20"} err="failed to get container status \"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20\": rpc error: code = NotFound desc = could not find container \"d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20\": container with ID starting with d375cafb7cdcd91ffeb2a4534e3108db2e3b8738ef484c2296d21feb94247f20 not found: ID does not exist"
Sep 16 18:48:36.519348 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.519324    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:48:36.524140 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:36.524117    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d884856bb-vvmk6"]
Sep 16 18:48:37.501477 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:37.501443    2578 generic.go:356] "Generic (PLEG): container finished" podID="37201f6a-b634-4878-80a4-f04088a883db" containerID="596b7b1625e5b39124df62621d19d1b45537377f8b8db50429a2c193ce8739d2" exitCode=0
Sep 16 18:48:37.501832 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:37.501513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vzbs8" event={"ID":"37201f6a-b634-4878-80a4-f04088a883db","Type":"ContainerDied","Data":"596b7b1625e5b39124df62621d19d1b45537377f8b8db50429a2c193ce8739d2"}
Sep 16 18:48:37.501832 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:37.501807    2578 scope.go:117] "RemoveContainer" containerID="596b7b1625e5b39124df62621d19d1b45537377f8b8db50429a2c193ce8739d2"
Sep 16 18:48:37.604456 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:37.604427    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554ca19e-2fbd-40af-b477-a35600489941" path="/var/lib/kubelet/pods/554ca19e-2fbd-40af-b477-a35600489941/volumes"
Sep 16 18:48:38.507718 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:38.507684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vzbs8" event={"ID":"37201f6a-b634-4878-80a4-f04088a883db","Type":"ContainerStarted","Data":"4497f918e0ce06363428f7ed896a55aec315097847d5d9e7e43a67ca2afb61e5"}
Sep 16 18:48:54.417427 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:54.417388    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:54.433338 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:54.433316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:54.591149 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:48:54.591119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:41.458312 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:50:41.458274    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:53:57.782011 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.781951    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qspfl"]
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554ca19e-2fbd-40af-b477-a35600489941" containerName="console"
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782277    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="554ca19e-2fbd-40af-b477-a35600489941" containerName="console"
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782296    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" containerName="registry"
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782301    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" containerName="registry"
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782349    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250f042-b8b5-4c5a-89ab-d0c94484a9f9" containerName="registry"
Sep 16 18:53:57.782465 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.782358    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="554ca19e-2fbd-40af-b477-a35600489941" containerName="console"
Sep 16 18:53:57.784600 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.784584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qspfl"
Sep 16 18:53:57.787639 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.787620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:57.787758 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.787620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:53:57.787758 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.787620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mv896\""
Sep 16 18:53:57.789045 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.789028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:57.789278 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.789260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:57.799554 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.799525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qspfl"]
Sep 16 18:53:57.969677 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.969641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:57.969677 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:57.969681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6sjh\" (UniqueName: \"kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.071021 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.070923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6sjh\" (UniqueName: \"kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.071162 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.071034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.071383 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.071367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.080925 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.080896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6sjh\" (UniqueName: \"kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh\") pod \"s3-init-qspfl\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") " pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.104304 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.104279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qspfl"
Sep 16 18:53:58.375048 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.375010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qspfl"]
Sep 16 18:53:58.377908 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:53:58.377876    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32f52297_3b6d_47a4_9bea_af6f98fbec8e.slice/crio-b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565 WatchSource:0}: Error finding container b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565: Status 404 returned error can't find the container with id b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565
Sep 16 18:53:58.379786 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.379764    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:53:58.730653 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:53:58.730574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerStarted","Data":"b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565"}
Sep 16 18:54:03.756214 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:54:03.756173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerStarted","Data":"6c6993e8335a85afa6ec0a98599b502978a780f17668eba20dc921c402903a7b"}
Sep 16 18:54:04.761939 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:54:04.761901    2578 generic.go:356] "Generic (PLEG): container finished" podID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerID="6c6993e8335a85afa6ec0a98599b502978a780f17668eba20dc921c402903a7b" exitCode=0
Sep 16 18:54:04.761939 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:54:04.761943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerDied","Data":"6c6993e8335a85afa6ec0a98599b502978a780f17668eba20dc921c402903a7b"}
Sep 16 18:54:09.786540 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:54:09.786507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerStarted","Data":"ad2d5e8dc6ffb457bbd3e16cf7cd7354277a26b9d45ea2535f3ea6864d3ec491"}
Sep 16 18:54:09.807843 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:54:09.807792    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qspfl" podStartSLOduration=2.632386162 podStartE2EDuration="12.807776864s" totalImagesPullingTime="8.774913443s" totalInitContainerRuntime="1.400477259s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:09.807321406 +0000 UTC m=+508.778798665" watchObservedRunningTime="2026-09-16 18:54:09.807776864 +0000 UTC m=+508.779254112"
Sep 16 18:55:15.033523 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:15.033487    2578 generic.go:356] "Generic (PLEG): container finished" podID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerID="ad2d5e8dc6ffb457bbd3e16cf7cd7354277a26b9d45ea2535f3ea6864d3ec491" exitCode=0
Sep 16 18:55:15.033945 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:15.033563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerDied","Data":"ad2d5e8dc6ffb457bbd3e16cf7cd7354277a26b9d45ea2535f3ea6864d3ec491"}
Sep 16 18:55:16.157635 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.157611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qspfl"
Sep 16 18:55:16.199790 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.199758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6sjh\" (UniqueName: \"kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh\") pod \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") "
Sep 16 18:55:16.199960 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.199898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache\") pod \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\" (UID: \"32f52297-3b6d-47a4-9bea-af6f98fbec8e\") "
Sep 16 18:55:16.202060 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.202028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh" (OuterVolumeSpecName: "kube-api-access-z6sjh") pod "32f52297-3b6d-47a4-9bea-af6f98fbec8e" (UID: "32f52297-3b6d-47a4-9bea-af6f98fbec8e"). InnerVolumeSpecName "kube-api-access-z6sjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:16.218146 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.218121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "32f52297-3b6d-47a4-9bea-af6f98fbec8e" (UID: "32f52297-3b6d-47a4-9bea-af6f98fbec8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:16.300650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.300579    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f52297-3b6d-47a4-9bea-af6f98fbec8e-model-cache\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:16.300650 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:16.300607    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6sjh\" (UniqueName: \"kubernetes.io/projected/32f52297-3b6d-47a4-9bea-af6f98fbec8e-kube-api-access-z6sjh\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:17.042581 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:17.042544    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qspfl"
Sep 16 18:55:17.042752 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:17.042547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qspfl" event={"ID":"32f52297-3b6d-47a4-9bea-af6f98fbec8e","Type":"ContainerDied","Data":"b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565"}
Sep 16 18:55:17.042752 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:17.042661    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40b970cb1bd38e0e5b9507c32f440d14e3ffdfd5dc4a78ce59e5ec02e032565"
Sep 16 18:55:28.246096 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246065    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-2mxn8"]
Sep 16 18:55:28.246526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerName="download-hf-model"
Sep 16 18:55:28.246526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerName="download-hf-model"
Sep 16 18:55:28.246526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerName="s3-init"
Sep 16 18:55:28.246526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerName="s3-init"
Sep 16 18:55:28.246526 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.246454    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f52297-3b6d-47a4-9bea-af6f98fbec8e" containerName="s3-init"
Sep 16 18:55:28.249656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.249636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.252515 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.252495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:28.252793 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.252775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:55:28.252903 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.252888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:28.252983 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.252920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:28.253884 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.253866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mv896\""
Sep 16 18:55:28.258574 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.258554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2mxn8"]
Sep 16 18:55:28.403039 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.402994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjzs\" (UniqueName: \"kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.403205 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.403109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.504243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.504154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjzs\" (UniqueName: \"kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.504243 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.504228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.504564 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.504548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.514009 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.513986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjzs\" (UniqueName: \"kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs\") pod \"s3-tls-init-custom-2mxn8\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") " pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.568973 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.568942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:28.841779 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:28.841751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2mxn8"]
Sep 16 18:55:28.843813 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:55:28.843783    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92f5f471_dbb9_4eb7_a347_57364e1cf06a.slice/crio-140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f WatchSource:0}: Error finding container 140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f: Status 404 returned error can't find the container with id 140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f
Sep 16 18:55:29.089311 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:29.089273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerStarted","Data":"e6f5fccce81aabdcf738dcca8a5a9db8a208e9fef947e1425f79a25495f6d6af"}
Sep 16 18:55:29.089495 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:29.089321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerStarted","Data":"140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f"}
Sep 16 18:55:31.099447 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:31.099412    2578 generic.go:356] "Generic (PLEG): container finished" podID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerID="e6f5fccce81aabdcf738dcca8a5a9db8a208e9fef947e1425f79a25495f6d6af" exitCode=0
Sep 16 18:55:31.099804 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:31.099490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerDied","Data":"e6f5fccce81aabdcf738dcca8a5a9db8a208e9fef947e1425f79a25495f6d6af"}
Sep 16 18:55:32.105335 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:32.105302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerStarted","Data":"68e8b3bdd0e5115cbb8b13b4fed337dafa2f86fbdc407b6de34a00323f3041ad"}
Sep 16 18:55:32.123687 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:32.123636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-2mxn8" podStartSLOduration=2.09683296 podStartE2EDuration="4.123618673s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.026785713s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:32.122609882 +0000 UTC m=+591.094087142" watchObservedRunningTime="2026-09-16 18:55:32.123618673 +0000 UTC m=+591.095095922"
Sep 16 18:55:37.127453 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:37.127420    2578 generic.go:356] "Generic (PLEG): container finished" podID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerID="68e8b3bdd0e5115cbb8b13b4fed337dafa2f86fbdc407b6de34a00323f3041ad" exitCode=0
Sep 16 18:55:37.127913 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:37.127499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerDied","Data":"68e8b3bdd0e5115cbb8b13b4fed337dafa2f86fbdc407b6de34a00323f3041ad"}
Sep 16 18:55:38.264302 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.264276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:38.387393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.387301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmjzs\" (UniqueName: \"kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs\") pod \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") "
Sep 16 18:55:38.387393 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.387370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache\") pod \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\" (UID: \"92f5f471-dbb9-4eb7-a347-57364e1cf06a\") "
Sep 16 18:55:38.389601 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.389564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs" (OuterVolumeSpecName: "kube-api-access-nmjzs") pod "92f5f471-dbb9-4eb7-a347-57364e1cf06a" (UID: "92f5f471-dbb9-4eb7-a347-57364e1cf06a"). InnerVolumeSpecName "kube-api-access-nmjzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:38.409575 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.409547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache" (OuterVolumeSpecName: "model-cache") pod "92f5f471-dbb9-4eb7-a347-57364e1cf06a" (UID: "92f5f471-dbb9-4eb7-a347-57364e1cf06a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:38.488227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.488197    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjzs\" (UniqueName: \"kubernetes.io/projected/92f5f471-dbb9-4eb7-a347-57364e1cf06a-kube-api-access-nmjzs\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.488227 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:38.488222    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f5f471-dbb9-4eb7-a347-57364e1cf06a-model-cache\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.136600 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:39.136563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2mxn8" event={"ID":"92f5f471-dbb9-4eb7-a347-57364e1cf06a","Type":"ContainerDied","Data":"140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f"}
Sep 16 18:55:39.136600 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:39.136580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2mxn8"
Sep 16 18:55:39.136600 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:39.136599    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="140f26003406b068a4a1f056274e3c34eac37e9953c26a63b23f6c1e6dd4e03f"
Sep 16 18:55:41.547013 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.546975    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-r9d4r"]
Sep 16 18:55:41.547367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.547291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerName="s3-tls-init-custom"
Sep 16 18:55:41.547367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.547304    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerName="s3-tls-init-custom"
Sep 16 18:55:41.547367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.547321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerName="download-hf-model"
Sep 16 18:55:41.547367 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.547329    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerName="download-hf-model"
Sep 16 18:55:41.547504 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.547369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f5f471-dbb9-4eb7-a347-57364e1cf06a" containerName="s3-tls-init-custom"
Sep 16 18:55:41.551348 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.551332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.554147 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.554124    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:41.554254 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.554124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mv896\""
Sep 16 18:55:41.554254 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.554169    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:55:41.555390 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.555372    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:41.555477 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.555451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:41.558562 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.558539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-r9d4r"]
Sep 16 18:55:41.715412 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.715374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.715590 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.715437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwr5\" (UniqueName: \"kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.816126 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.816042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.816258 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.816125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwr5\" (UniqueName: \"kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.816484 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.816464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.825073 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.825054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwr5\" (UniqueName: \"kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5\") pod \"s3-tls-init-serving-r9d4r\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") " pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:41.872769 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:41.872741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:42.144647 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:42.144614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-r9d4r"]
Sep 16 18:55:42.148357 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:55:42.148329    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c916d7_6c42_4a06_b44a_a3591e976453.slice/crio-9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368 WatchSource:0}: Error finding container 9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368: Status 404 returned error can't find the container with id 9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368
Sep 16 18:55:43.153312 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:43.153273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerStarted","Data":"5072a7381fd4032517e92f689ecb7cea06d0ca05b45d97f9b95687304d0370a5"}
Sep 16 18:55:43.153312 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:43.153318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerStarted","Data":"9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368"}
Sep 16 18:55:44.158534 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:44.158490    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerID="5072a7381fd4032517e92f689ecb7cea06d0ca05b45d97f9b95687304d0370a5" exitCode=0
Sep 16 18:55:44.158941 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:44.158534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerDied","Data":"5072a7381fd4032517e92f689ecb7cea06d0ca05b45d97f9b95687304d0370a5"}
Sep 16 18:55:45.164992 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:45.164956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerStarted","Data":"f1bfc0c6b161e1e737aaf0e0bccc3f6ae8907a0889aba65f496cfd5119f518ed"}
Sep 16 18:55:45.184343 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:45.184295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-r9d4r" podStartSLOduration=2.471734257 podStartE2EDuration="4.184277996s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.712543739s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:45.182719049 +0000 UTC m=+604.154196315" watchObservedRunningTime="2026-09-16 18:55:45.184277996 +0000 UTC m=+604.155755244"
Sep 16 18:55:52.198329 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:52.198296    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerID="f1bfc0c6b161e1e737aaf0e0bccc3f6ae8907a0889aba65f496cfd5119f518ed" exitCode=0
Sep 16 18:55:52.198741 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:52.198368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerDied","Data":"f1bfc0c6b161e1e737aaf0e0bccc3f6ae8907a0889aba65f496cfd5119f518ed"}
Sep 16 18:55:53.330007 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.329975    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:53.515291 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.515207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache\") pod \"c9c916d7-6c42-4a06-b44a-a3591e976453\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") "
Sep 16 18:55:53.515291 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.515243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wwr5\" (UniqueName: \"kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5\") pod \"c9c916d7-6c42-4a06-b44a-a3591e976453\" (UID: \"c9c916d7-6c42-4a06-b44a-a3591e976453\") "
Sep 16 18:55:53.517505 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.517479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5" (OuterVolumeSpecName: "kube-api-access-7wwr5") pod "c9c916d7-6c42-4a06-b44a-a3591e976453" (UID: "c9c916d7-6c42-4a06-b44a-a3591e976453"). InnerVolumeSpecName "kube-api-access-7wwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:53.538081 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.538045    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9c916d7-6c42-4a06-b44a-a3591e976453" (UID: "c9c916d7-6c42-4a06-b44a-a3591e976453"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:53.615886 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.615864    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c916d7-6c42-4a06-b44a-a3591e976453-model-cache\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.615970 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:53.615888    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwr5\" (UniqueName: \"kubernetes.io/projected/c9c916d7-6c42-4a06-b44a-a3591e976453-kube-api-access-7wwr5\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:55:54.207260 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:54.207235    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r9d4r"
Sep 16 18:55:54.207260 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:54.207246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r9d4r" event={"ID":"c9c916d7-6c42-4a06-b44a-a3591e976453","Type":"ContainerDied","Data":"9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368"}
Sep 16 18:55:54.207478 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:55:54.207285    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5d6db288a64fdfd755256bb6c77e3f04818be791d95236f7d6ce74144b2368"
Sep 16 18:56:08.003122 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003042    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:08.003493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerName="download-hf-model"
Sep 16 18:56:08.003493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003427    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerName="download-hf-model"
Sep 16 18:56:08.003493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerName="s3-tls-init-serving"
Sep 16 18:56:08.003493 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003448    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerName="s3-tls-init-serving"
Sep 16 18:56:08.003616 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.003499    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c916d7-6c42-4a06-b44a-a3591e976453" containerName="s3-tls-init-serving"
Sep 16 18:56:08.006544 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.006528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.009158 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.009136    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:08.009462 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.009442    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:08.010553 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.010531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a79348-predictor-serving-cert\""
Sep 16 18:56:08.010656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.010573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a79348-kube-rbac-proxy-sar-config\""
Sep 16 18:56:08.010656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.010533    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a79348-dockercfg-g8kkx\""
Sep 16 18:56:08.010656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.010536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:08.010656 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.010535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a79348\""
Sep 16 18:56:08.018408 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.018388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:08.022631 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.022611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.022728 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.022651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.022845 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.022743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rkj\" (UniqueName: \"kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.022845 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.022775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.022845 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.022805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124055 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rkj\" (UniqueName: \"kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124055 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124259 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124259 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124259 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124607 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124835 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.124960 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.124821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.126697 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.126681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.134357 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.134336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rkj\" (UniqueName: \"kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj\") pod \"isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") " pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.317051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.316969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:08.593705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:08.593648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:08.596653 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:56:08.596630    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55acadb9_634c_4939_9198_f8716656d632.slice/crio-1c1d66d64715db6c2d4656b32b923be4d32709eaccb41fd06011c49a08d71f0a WatchSource:0}: Error finding container 1c1d66d64715db6c2d4656b32b923be4d32709eaccb41fd06011c49a08d71f0a: Status 404 returned error can't find the container with id 1c1d66d64715db6c2d4656b32b923be4d32709eaccb41fd06011c49a08d71f0a
Sep 16 18:56:09.276183 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:09.276141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerStarted","Data":"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"}
Sep 16 18:56:09.276559 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:09.276195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerStarted","Data":"1c1d66d64715db6c2d4656b32b923be4d32709eaccb41fd06011c49a08d71f0a"}
Sep 16 18:56:14.299594 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:14.299560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/0.log"
Sep 16 18:56:14.299981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:14.299600    2578 generic.go:356] "Generic (PLEG): container finished" podID="55acadb9-634c-4939-9198-f8716656d632" containerID="4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb" exitCode=1
Sep 16 18:56:14.299981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:14.299677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerDied","Data":"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"}
Sep 16 18:56:15.305154 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:15.305127    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/0.log"
Sep 16 18:56:15.305561 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:15.305235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerStarted","Data":"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"}
Sep 16 18:56:17.686894 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:17.686842    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:17.687318 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:17.687115    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer" containerID="cri-o://46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e" gracePeriod=30
Sep 16 18:56:18.922104 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:18.922081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/1.log"
Sep 16 18:56:18.922548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:18.922533    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/0.log"
Sep 16 18:56:18.924887 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:18.924864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:19.010588 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert\") pod \"55acadb9-634c-4939-9198-f8716656d632\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") "
Sep 16 18:56:19.010753 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location\") pod \"55acadb9-634c-4939-9198-f8716656d632\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") "
Sep 16 18:56:19.010753 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4rkj\" (UniqueName: \"kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj\") pod \"55acadb9-634c-4939-9198-f8716656d632\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") "
Sep 16 18:56:19.010753 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config\") pod \"55acadb9-634c-4939-9198-f8716656d632\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") "
Sep 16 18:56:19.010753 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls\") pod \"55acadb9-634c-4939-9198-f8716656d632\" (UID: \"55acadb9-634c-4939-9198-f8716656d632\") "
Sep 16 18:56:19.010979 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55acadb9-634c-4939-9198-f8716656d632" (UID: "55acadb9-634c-4939-9198-f8716656d632"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:19.011029 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55acadb9-634c-4939-9198-f8716656d632" (UID: "55acadb9-634c-4939-9198-f8716656d632"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:19.011029 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.010999    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acadb9-634c-4939-9198-f8716656d632-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.011128 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.011103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55acadb9-634c-4939-9198-f8716656d632" (UID: "55acadb9-634c-4939-9198-f8716656d632"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:19.012870 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.012836    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55acadb9-634c-4939-9198-f8716656d632" (UID: "55acadb9-634c-4939-9198-f8716656d632"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:19.013026 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.013007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj" (OuterVolumeSpecName: "kube-api-access-d4rkj") pod "55acadb9-634c-4939-9198-f8716656d632" (UID: "55acadb9-634c-4939-9198-f8716656d632"). InnerVolumeSpecName "kube-api-access-d4rkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:19.111691 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.111660    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.111691 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.111688    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acadb9-634c-4939-9198-f8716656d632-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.111836 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.111699    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acadb9-634c-4939-9198-f8716656d632-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.111836 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.111707    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rkj\" (UniqueName: \"kubernetes.io/projected/55acadb9-634c-4939-9198-f8716656d632-kube-api-access-d4rkj\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.329908 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.329814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/1.log"
Sep 16 18:56:19.330220 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl_55acadb9-634c-4939-9198-f8716656d632/storage-initializer/0.log"
Sep 16 18:56:19.330318 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330244    2578 generic.go:356] "Generic (PLEG): container finished" podID="55acadb9-634c-4939-9198-f8716656d632" containerID="46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e" exitCode=1
Sep 16 18:56:19.330318 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330302    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"
Sep 16 18:56:19.330407 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerDied","Data":"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"}
Sep 16 18:56:19.330407 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl" event={"ID":"55acadb9-634c-4939-9198-f8716656d632","Type":"ContainerDied","Data":"1c1d66d64715db6c2d4656b32b923be4d32709eaccb41fd06011c49a08d71f0a"}
Sep 16 18:56:19.330477 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.330415    2578 scope.go:117] "RemoveContainer" containerID="46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"
Sep 16 18:56:19.339305 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.339290    2578 scope.go:117] "RemoveContainer" containerID="4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"
Sep 16 18:56:19.346593 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.346577    2578 scope.go:117] "RemoveContainer" containerID="46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"
Sep 16 18:56:19.346957 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:56:19.346829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e\": container with ID starting with 46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e not found: ID does not exist" containerID="46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"
Sep 16 18:56:19.346957 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.346887    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e"} err="failed to get container status \"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e\": rpc error: code = NotFound desc = could not find container \"46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e\": container with ID starting with 46b0cba253a67634efd8791755063bab98f068246813eeb053684c9f14919a2e not found: ID does not exist"
Sep 16 18:56:19.346957 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.346910    2578 scope.go:117] "RemoveContainer" containerID="4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"
Sep 16 18:56:19.347160 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:56:19.347144    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb\": container with ID starting with 4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb not found: ID does not exist" containerID="4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"
Sep 16 18:56:19.347212 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.347168    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb"} err="failed to get container status \"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb\": rpc error: code = NotFound desc = could not find container \"4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb\": container with ID starting with 4ebd74a1b3f4c6cb5ee91ff0f7549fcf3ce8c534e9759c5d6cf10aad4d5a4aeb not found: ID does not exist"
Sep 16 18:56:19.373167 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.373144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:19.378447 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.378411    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a79348-predictor-7f56f9566b-dfkpl"]
Sep 16 18:56:19.602253 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:19.602222    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55acadb9-634c-4939-9198-f8716656d632" path="/var/lib/kubelet/pods/55acadb9-634c-4939-9198-f8716656d632/volumes"
Sep 16 18:56:22.869692 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.869659    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.869971    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.869983    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.869994    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.869999    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.870068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872051 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.870153    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55acadb9-634c-4939-9198-f8716656d632" containerName="storage-initializer"
Sep 16 18:56:22.872948 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.872933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:22.877981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.877954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 18:56:22.877981 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.877965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:22.878251 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.878234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 18:56:22.878308 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.878240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:22.878353 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.878257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:22.878814 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.878796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 18:56:22.878882 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.878817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:22.884934 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.884914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:56:22.942816 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.942781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:22.942990 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.942838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:22.942990 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.942952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:22.943065 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.942990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgj2\" (UniqueName: \"kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:22.943065 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:22.943008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.043749 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.043718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.043958 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.043759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgj2\" (UniqueName: \"kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.043958 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.043779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.043958 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.043797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.043958 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.043841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.044336 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.044309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.044634 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.044610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.044692 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.044675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.046457 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.046411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.053714 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.053694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgj2\" (UniqueName: \"kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-dpngq\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.182491 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.182422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:23.454121 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:23.454097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:56:23.456207 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:56:23.456180    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19450878_e5a6_48c5_8153_10d26b9abfa5.slice/crio-85c43fe941ef75b90852aec01767f64cf050036f6154d5c94db9f89edd6b7883 WatchSource:0}: Error finding container 85c43fe941ef75b90852aec01767f64cf050036f6154d5c94db9f89edd6b7883: Status 404 returned error can't find the container with id 85c43fe941ef75b90852aec01767f64cf050036f6154d5c94db9f89edd6b7883
Sep 16 18:56:24.353105 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:24.353066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerStarted","Data":"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"}
Sep 16 18:56:24.353476 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:24.353137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerStarted","Data":"85c43fe941ef75b90852aec01767f64cf050036f6154d5c94db9f89edd6b7883"}
Sep 16 18:56:28.371327 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:28.371287    2578 generic.go:356] "Generic (PLEG): container finished" podID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerID="9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6" exitCode=0
Sep 16 18:56:28.371706 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:28.371358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerDied","Data":"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"}
Sep 16 18:56:50.491424 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:50.491381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerStarted","Data":"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"}
Sep 16 18:56:53.508619 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:53.508531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerStarted","Data":"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"}
Sep 16 18:56:53.509076 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:53.508641    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:53.531321 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:53.531270    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podStartSLOduration=2.833159529 podStartE2EDuration="31.531254682s" totalImagesPullingTime="24.721008237s" totalInitContainerRuntime="3.977086916s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:53.529311824 +0000 UTC m=+672.500789095" watchObservedRunningTime="2026-09-16 18:56:53.531254682 +0000 UTC m=+672.502731946"
Sep 16 18:56:54.512708 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:54.512666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:56:54.514039 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:54.514009    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:56:55.516738 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:56:55.516696    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:00.520633 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:00.520603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:57:00.521274 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:00.521246    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:10.521837 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:10.521795    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:20.521681 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:20.521640    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:30.521244 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:30.521204    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:40.522078 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:40.521999    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:57:50.522108 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:57:50.522067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:58:00.521201 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:00.521158    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:58:07.602828 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:07.602801    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:58:12.954269 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:12.954236    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:58:12.954670 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:12.954506    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container" containerID="cri-o://2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea" gracePeriod=30
Sep 16 18:58:12.954670 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:12.954520    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kube-rbac-proxy" containerID="cri-o://4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462" gracePeriod=30
Sep 16 18:58:13.828485 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:13.828451    2578 generic.go:356] "Generic (PLEG): container finished" podID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerID="4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462" exitCode=2
Sep 16 18:58:13.828652 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:13.828521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerDied","Data":"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"}
Sep 16 18:58:15.517122 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:15.517083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 16 18:58:17.590128 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.590101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:58:17.732638 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location\") pod \"19450878-e5a6-48c5-8153-10d26b9abfa5\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") "
Sep 16 18:58:17.732638 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcgj2\" (UniqueName: \"kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2\") pod \"19450878-e5a6-48c5-8153-10d26b9abfa5\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") "
Sep 16 18:58:17.732886 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert\") pod \"19450878-e5a6-48c5-8153-10d26b9abfa5\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") "
Sep 16 18:58:17.732886 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config\") pod \"19450878-e5a6-48c5-8153-10d26b9abfa5\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") "
Sep 16 18:58:17.732886 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls\") pod \"19450878-e5a6-48c5-8153-10d26b9abfa5\" (UID: \"19450878-e5a6-48c5-8153-10d26b9abfa5\") "
Sep 16 18:58:17.733045 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.732929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19450878-e5a6-48c5-8153-10d26b9abfa5" (UID: "19450878-e5a6-48c5-8153-10d26b9abfa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:17.733045 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.733024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19450878-e5a6-48c5-8153-10d26b9abfa5" (UID: "19450878-e5a6-48c5-8153-10d26b9abfa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:17.733145 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.733121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19450878-e5a6-48c5-8153-10d26b9abfa5" (UID: "19450878-e5a6-48c5-8153-10d26b9abfa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:17.734943 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.734920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2" (OuterVolumeSpecName: "kube-api-access-qcgj2") pod "19450878-e5a6-48c5-8153-10d26b9abfa5" (UID: "19450878-e5a6-48c5-8153-10d26b9abfa5"). InnerVolumeSpecName "kube-api-access-qcgj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:17.735026 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.734996    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19450878-e5a6-48c5-8153-10d26b9abfa5" (UID: "19450878-e5a6-48c5-8153-10d26b9abfa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:17.833235 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.833196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.833235 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.833230    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19450878-e5a6-48c5-8153-10d26b9abfa5-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.833235 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.833241    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19450878-e5a6-48c5-8153-10d26b9abfa5-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.833235 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.833252    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcgj2\" (UniqueName: \"kubernetes.io/projected/19450878-e5a6-48c5-8153-10d26b9abfa5-kube-api-access-qcgj2\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.833481 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.833261    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19450878-e5a6-48c5-8153-10d26b9abfa5-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.849007 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.848979    2578 generic.go:356] "Generic (PLEG): container finished" podID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerID="2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea" exitCode=0
Sep 16 18:58:17.849131 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.849028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerDied","Data":"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"}
Sep 16 18:58:17.849131 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.849039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"
Sep 16 18:58:17.849131 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.849074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq" event={"ID":"19450878-e5a6-48c5-8153-10d26b9abfa5","Type":"ContainerDied","Data":"85c43fe941ef75b90852aec01767f64cf050036f6154d5c94db9f89edd6b7883"}
Sep 16 18:58:17.849131 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.849101    2578 scope.go:117] "RemoveContainer" containerID="4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"
Sep 16 18:58:17.858232 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.858213    2578 scope.go:117] "RemoveContainer" containerID="2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"
Sep 16 18:58:17.865737 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.865718    2578 scope.go:117] "RemoveContainer" containerID="9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"
Sep 16 18:58:17.872188 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.872165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:58:17.873223 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873210    2578 scope.go:117] "RemoveContainer" containerID="4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"
Sep 16 18:58:17.873486 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:58:17.873468    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462\": container with ID starting with 4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462 not found: ID does not exist" containerID="4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"
Sep 16 18:58:17.873548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873499    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462"} err="failed to get container status \"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462\": rpc error: code = NotFound desc = could not find container \"4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462\": container with ID starting with 4f587a8236c0b605734a850639fe6af78191925109c2bf58a8ee6509193a2462 not found: ID does not exist"
Sep 16 18:58:17.873548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873514    2578 scope.go:117] "RemoveContainer" containerID="2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"
Sep 16 18:58:17.873728 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:58:17.873712    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea\": container with ID starting with 2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea not found: ID does not exist" containerID="2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"
Sep 16 18:58:17.873770 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873731    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea"} err="failed to get container status \"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea\": rpc error: code = NotFound desc = could not find container \"2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea\": container with ID starting with 2baffee7f548e48cd0160f864a00d5e8ba2d1a67cf9466110442c459b03594ea not found: ID does not exist"
Sep 16 18:58:17.873770 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873744    2578 scope.go:117] "RemoveContainer" containerID="9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"
Sep 16 18:58:17.873987 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:58:17.873969    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6\": container with ID starting with 9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6 not found: ID does not exist" containerID="9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"
Sep 16 18:58:17.874086 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.873991    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6"} err="failed to get container status \"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6\": rpc error: code = NotFound desc = could not find container \"9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6\": container with ID starting with 9aa81470219fd88aa8c547619b1af7f2327f33ce33ffed987d1aa0acdf281fb6 not found: ID does not exist"
Sep 16 18:58:17.876172 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:17.876141    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-dpngq"]
Sep 16 18:58:18.163574 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163539    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:58:18.163872 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163836    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container"
Sep 16 18:58:18.163872 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163863    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163877    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kube-rbac-proxy"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kube-rbac-proxy"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="storage-initializer"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="storage-initializer"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163943    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kserve-container"
Sep 16 18:58:18.163962 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.163952    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" containerName="kube-rbac-proxy"
Sep 16 18:58:18.166827 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.166810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.169753 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.169726    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:18.170259 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.170202    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 18:58:18.170364 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.170347    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:58:18.170532 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.170486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 18:58:18.170841 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.170826    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:58:18.171266 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.171248    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 18:58:18.171314 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.171299    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:58:18.185078 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.185053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:58:18.236201 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.236153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.236381 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.236207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42kp\" (UniqueName: \"kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.236381 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.236291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.236381 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.236370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.236520 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.236398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337445 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42kp\" (UniqueName: \"kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337611 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337611 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337611 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337611 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.337796 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:58:18.337652    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 16 18:58:18.337796 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:58:18.337734    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls podName:065a5044-1dfc-4905-a4df-bcfaa093f83e nodeName:}" failed. No retries permitted until 2026-09-16 18:58:18.837714041 +0000 UTC m=+757.809191281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-6xt54" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 16 18:58:18.337948 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.337931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.338342 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.338321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.338388 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.338321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.349904 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.349886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42kp\" (UniqueName: \"kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.842189 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.842150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:18.844616 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:18.844596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-6xt54\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:19.077262 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:19.077234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:19.355349 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:19.355323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:58:19.357693 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:58:19.357662    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a5044_1dfc_4905_a4df_bcfaa093f83e.slice/crio-e5ce99e96921db84bd240eb0ab162d8c6f668baa0946dc51f15a637be61fa3ce WatchSource:0}: Error finding container e5ce99e96921db84bd240eb0ab162d8c6f668baa0946dc51f15a637be61fa3ce: Status 404 returned error can't find the container with id e5ce99e96921db84bd240eb0ab162d8c6f668baa0946dc51f15a637be61fa3ce
Sep 16 18:58:19.605219 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:19.605182    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19450878-e5a6-48c5-8153-10d26b9abfa5" path="/var/lib/kubelet/pods/19450878-e5a6-48c5-8153-10d26b9abfa5/volumes"
Sep 16 18:58:19.859092 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:19.859045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerStarted","Data":"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"}
Sep 16 18:58:19.859457 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:19.859150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerStarted","Data":"e5ce99e96921db84bd240eb0ab162d8c6f668baa0946dc51f15a637be61fa3ce"}
Sep 16 18:58:23.875642 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:23.875605    2578 generic.go:356] "Generic (PLEG): container finished" podID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerID="7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8" exitCode=0
Sep 16 18:58:23.876146 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:23.875681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerDied","Data":"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"}
Sep 16 18:58:24.881863 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:24.881819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerStarted","Data":"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"}
Sep 16 18:58:24.882201 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:24.881904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerStarted","Data":"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"}
Sep 16 18:58:24.882201 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:24.882097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:24.906202 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:24.906158    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podStartSLOduration=3.133910659 podStartE2EDuration="6.90614269s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.772232031s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:24.904238159 +0000 UTC m=+763.875715408" watchObservedRunningTime="2026-09-16 18:58:24.90614269 +0000 UTC m=+763.877619939"
Sep 16 18:58:25.885751 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:25.885708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:25.887136 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:25.887107    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:58:26.890026 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:26.889978    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:58:31.895597 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:31.895568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:58:31.896214 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:31.896015    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:58:41.896654 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:41.896612    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:58:51.896610 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:58:51.896570    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:01.897005 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:01.896966    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:11.896119 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:11.896033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:18.086548 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.086515    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 18:59:18.089791 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.089772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.093033 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.093012    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 16 18:59:18.093600 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.093579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 16 18:59:18.105559 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.105535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 18:59:18.235781 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.235746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.235940 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.235793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.235940 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.235923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmg6x\" (UniqueName: \"kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.336421 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.336386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.336618 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.336491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmg6x\" (UniqueName: \"kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.336618 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.336533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.337069 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.337045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.338991 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.338971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.347816 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.347795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmg6x\" (UniqueName: \"kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x\") pod \"message-dumper-predictor-6575bcfdfd-7kkj5\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.400009 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.399978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:18.671521 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.671475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 18:59:18.673744 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:59:18.673717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f69884_7f94_4674_82bf_7b70f5a955d9.slice/crio-e08cf5da2eebd59d271b6a3c70b1f9f00871c8aa787ab0357a4d9908601599a7 WatchSource:0}: Error finding container e08cf5da2eebd59d271b6a3c70b1f9f00871c8aa787ab0357a4d9908601599a7: Status 404 returned error can't find the container with id e08cf5da2eebd59d271b6a3c70b1f9f00871c8aa787ab0357a4d9908601599a7
Sep 16 18:59:18.675705 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:18.675688    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:59:19.097403 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:19.097373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerStarted","Data":"e08cf5da2eebd59d271b6a3c70b1f9f00871c8aa787ab0357a4d9908601599a7"}
Sep 16 18:59:20.103278 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.103248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerStarted","Data":"8dff69e25bb68832b1b3dcd10f3986de5c9faed32f5d759bb73778a5788dc593"}
Sep 16 18:59:20.103632 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.103307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerStarted","Data":"ac54ecb6df726ce7089de836399e9a868f8a0ca217824c496ffdd6f40efc44ae"}
Sep 16 18:59:20.103632 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.103482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:20.103632 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.103607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:20.105000 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.104986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:20.122718 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:20.122675    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" podStartSLOduration=0.983092095 podStartE2EDuration="2.122662803s" totalImagesPullingTime="1.139570708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:20.121479096 +0000 UTC m=+819.092956358" watchObservedRunningTime="2026-09-16 18:59:20.122662803 +0000 UTC m=+819.094140051"
Sep 16 18:59:21.896268 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:21.896229    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:27.116399 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:27.116369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 18:59:31.896481 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:31.896431    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:41.896997 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:41.896969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:59:48.236821 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:48.236772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:59:48.237398 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:48.237139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" containerID="cri-o://068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82" gracePeriod=30
Sep 16 18:59:48.237398 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:48.237247    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kube-rbac-proxy" containerID="cri-o://cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172" gracePeriod=30
Sep 16 18:59:49.225382 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:49.225348    2578 generic.go:356] "Generic (PLEG): container finished" podID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerID="cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172" exitCode=2
Sep 16 18:59:49.225550 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:49.225405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerDied","Data":"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"}
Sep 16 18:59:51.890672 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:51.890624    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 16 18:59:51.896003 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:51.895977    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:59:52.884313 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:52.884291    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:59:53.017179 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config\") pod \"065a5044-1dfc-4905-a4df-bcfaa093f83e\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") "
Sep 16 18:59:53.017529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") pod \"065a5044-1dfc-4905-a4df-bcfaa093f83e\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") "
Sep 16 18:59:53.017529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location\") pod \"065a5044-1dfc-4905-a4df-bcfaa093f83e\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") "
Sep 16 18:59:53.017529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b42kp\" (UniqueName: \"kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp\") pod \"065a5044-1dfc-4905-a4df-bcfaa093f83e\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") "
Sep 16 18:59:53.017529 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert\") pod \"065a5044-1dfc-4905-a4df-bcfaa093f83e\" (UID: \"065a5044-1dfc-4905-a4df-bcfaa093f83e\") "
Sep 16 18:59:53.017719 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "065a5044-1dfc-4905-a4df-bcfaa093f83e" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:53.017719 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "065a5044-1dfc-4905-a4df-bcfaa093f83e" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:53.017796 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.017712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "065a5044-1dfc-4905-a4df-bcfaa093f83e" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:53.019480 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.019456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp" (OuterVolumeSpecName: "kube-api-access-b42kp") pod "065a5044-1dfc-4905-a4df-bcfaa093f83e" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e"). InnerVolumeSpecName "kube-api-access-b42kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:53.019480 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.019473    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "065a5044-1dfc-4905-a4df-bcfaa093f83e" (UID: "065a5044-1dfc-4905-a4df-bcfaa093f83e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:53.118152 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.118123    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42kp\" (UniqueName: \"kubernetes.io/projected/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-api-access-b42kp\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.118152 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.118149    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.118152 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.118158    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065a5044-1dfc-4905-a4df-bcfaa093f83e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.118342 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.118167    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065a5044-1dfc-4905-a4df-bcfaa093f83e-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.118342 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.118175    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065a5044-1dfc-4905-a4df-bcfaa093f83e-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.244417 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.244381    2578 generic.go:356] "Generic (PLEG): container finished" podID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerID="068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82" exitCode=0
Sep 16 18:59:53.244562 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.244472    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"
Sep 16 18:59:53.244562 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.244468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerDied","Data":"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"}
Sep 16 18:59:53.244646 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.244633    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54" event={"ID":"065a5044-1dfc-4905-a4df-bcfaa093f83e","Type":"ContainerDied","Data":"e5ce99e96921db84bd240eb0ab162d8c6f668baa0946dc51f15a637be61fa3ce"}
Sep 16 18:59:53.244680 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.244664    2578 scope.go:117] "RemoveContainer" containerID="cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"
Sep 16 18:59:53.254134 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.254097    2578 scope.go:117] "RemoveContainer" containerID="068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"
Sep 16 18:59:53.264663 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.264635    2578 scope.go:117] "RemoveContainer" containerID="7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"
Sep 16 18:59:53.272249 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.272186    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:59:53.274358 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.274338    2578 scope.go:117] "RemoveContainer" containerID="cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"
Sep 16 18:59:53.274647 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:59:53.274625    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172\": container with ID starting with cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172 not found: ID does not exist" containerID="cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"
Sep 16 18:59:53.274733 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.274659    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172"} err="failed to get container status \"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172\": rpc error: code = NotFound desc = could not find container \"cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172\": container with ID starting with cc2956d7a739b6ad86ac431b3efeac7a0f9b70db18479b4a3017a7992628b172 not found: ID does not exist"
Sep 16 18:59:53.274733 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.274683    2578 scope.go:117] "RemoveContainer" containerID="068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"
Sep 16 18:59:53.274991 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:59:53.274971    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82\": container with ID starting with 068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82 not found: ID does not exist" containerID="068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"
Sep 16 18:59:53.275070 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.274998    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82"} err="failed to get container status \"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82\": rpc error: code = NotFound desc = could not find container \"068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82\": container with ID starting with 068e4bd3fac43c6c37aaa697ba83b326ced96238ec1c13f2fab4a649a1152f82 not found: ID does not exist"
Sep 16 18:59:53.275070 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.275030    2578 scope.go:117] "RemoveContainer" containerID="7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"
Sep 16 18:59:53.275302 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:59:53.275281    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8\": container with ID starting with 7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8 not found: ID does not exist" containerID="7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"
Sep 16 18:59:53.275392 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.275305    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8"} err="failed to get container status \"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8\": rpc error: code = NotFound desc = could not find container \"7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8\": container with ID starting with 7bb7e0e1682393100c6e8af74e5c37ef05413db364e56df8462eb718eadb88c8 not found: ID does not exist"
Sep 16 18:59:53.280342 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.280308    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-6xt54"]
Sep 16 18:59:53.443263 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443230    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 18:59:53.443546 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443534    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="storage-initializer"
Sep 16 18:59:53.443589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443547    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="storage-initializer"
Sep 16 18:59:53.443589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container"
Sep 16 18:59:53.443589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container"
Sep 16 18:59:53.443589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kube-rbac-proxy"
Sep 16 18:59:53.443589 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443584    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kube-rbac-proxy"
Sep 16 18:59:53.443824 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443642    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kserve-container"
Sep 16 18:59:53.443824 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.443650    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" containerName="kube-rbac-proxy"
Sep 16 18:59:53.448488 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.448470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.451408 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.451388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 18:59:53.454017 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.453813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 18:59:53.454017 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.453934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:53.454017 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.453964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:53.458807 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.458782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 18:59:53.522537 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.522467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.522537 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.522514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.522687 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.522561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6ng\" (UniqueName: \"kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.522687 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.522616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.522687 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.522636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.603361 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.603334    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065a5044-1dfc-4905-a4df-bcfaa093f83e" path="/var/lib/kubelet/pods/065a5044-1dfc-4905-a4df-bcfaa093f83e/volumes"
Sep 16 18:59:53.623450 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.623535 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.623535 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.623535 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.623680 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6ng\" (UniqueName: \"kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.623680 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:59:53.623631    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:59:53.623783 ip-10-0-142-130 kubenswrapper[2578]: E0916 18:59:53.623713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls podName:a6f7a3e1-4b83-4ba2-97e2-135f230f1768 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:54.123689867 +0000 UTC m=+853.095167110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:59:53.623918 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.623899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.624143 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.624125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.624192 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.624165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:53.632358 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:53.632332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6ng\" (UniqueName: \"kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:54.128234 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:54.128200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:54.130830 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:54.130805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:54.359374 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:54.359321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 18:59:54.633412 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:54.633386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 18:59:54.635424 ip-10-0-142-130 kubenswrapper[2578]: W0916 18:59:54.635395    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f7a3e1_4b83_4ba2_97e2_135f230f1768.slice/crio-4979c8e85694fbf883ea87504fb258ee1ac6acf768d18c0cebe47eb7625af1d8 WatchSource:0}: Error finding container 4979c8e85694fbf883ea87504fb258ee1ac6acf768d18c0cebe47eb7625af1d8: Status 404 returned error can't find the container with id 4979c8e85694fbf883ea87504fb258ee1ac6acf768d18c0cebe47eb7625af1d8
Sep 16 18:59:55.255464 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:55.255429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerStarted","Data":"2cf5fe8228f7b7dea17a93541cb404220533c97e115bc1e29e0d2571defa21ab"}
Sep 16 18:59:55.255833 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:55.255485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerStarted","Data":"4979c8e85694fbf883ea87504fb258ee1ac6acf768d18c0cebe47eb7625af1d8"}
Sep 16 18:59:59.274460 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:59.274423    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerID="2cf5fe8228f7b7dea17a93541cb404220533c97e115bc1e29e0d2571defa21ab" exitCode=0
Sep 16 18:59:59.274820 ip-10-0-142-130 kubenswrapper[2578]: I0916 18:59:59.274497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerDied","Data":"2cf5fe8228f7b7dea17a93541cb404220533c97e115bc1e29e0d2571defa21ab"}
Sep 16 19:00:00.280983 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.280949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerStarted","Data":"9056f4c98d74ee18ec430b3ce1107ae90c08ca364bf5748779efe753f25b571f"}
Sep 16 19:00:00.281363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.281010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerStarted","Data":"a21cbebde8d284affd7e4fec70267eeede4acf5c5da7e1dd8052fd1164171af1"}
Sep 16 19:00:00.281363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.281331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:00:00.281478 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.281460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:00:00.282693 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.282671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:00.303095 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:00.303047    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podStartSLOduration=3.55251811 podStartE2EDuration="7.303028751s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750510641s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:00.301774855 +0000 UTC m=+859.273252105" watchObservedRunningTime="2026-09-16 19:00:00.303028751 +0000 UTC m=+859.274506001"
Sep 16 19:00:01.285351 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:01.285310    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:02.289172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:02.289126    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:07.293908 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:07.293877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:00:07.294360 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:07.294336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:17.294436 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:17.294396    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:27.294861 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:27.294818    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:37.294834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:37.294748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:47.295297 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:47.295256    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:00:53.136510 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.136478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-7kkj5_44f69884-7f94-4674-82bf-7b70f5a955d9/kserve-container/0.log"
Sep 16 19:00:53.342161 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.342130    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 19:00:53.342379 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.342354    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kserve-container" containerID="cri-o://ac54ecb6df726ce7089de836399e9a868f8a0ca217824c496ffdd6f40efc44ae" gracePeriod=30
Sep 16 19:00:53.342515 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.342457    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kube-rbac-proxy" containerID="cri-o://8dff69e25bb68832b1b3dcd10f3986de5c9faed32f5d759bb73778a5788dc593" gracePeriod=30
Sep 16 19:00:53.459185 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.459157    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:00:53.463363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.463342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.466613 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.466577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 19:00:53.466732 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.466624    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:00:53.478105 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.478049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:00:53.507554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.507520    2578 generic.go:356] "Generic (PLEG): container finished" podID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerID="8dff69e25bb68832b1b3dcd10f3986de5c9faed32f5d759bb73778a5788dc593" exitCode=2
Sep 16 19:00:53.507554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.507550    2578 generic.go:356] "Generic (PLEG): container finished" podID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerID="ac54ecb6df726ce7089de836399e9a868f8a0ca217824c496ffdd6f40efc44ae" exitCode=2
Sep 16 19:00:53.507768 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.507588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerDied","Data":"8dff69e25bb68832b1b3dcd10f3986de5c9faed32f5d759bb73778a5788dc593"}
Sep 16 19:00:53.507768 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.507639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerDied","Data":"ac54ecb6df726ce7089de836399e9a868f8a0ca217824c496ffdd6f40efc44ae"}
Sep 16 19:00:53.517165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.517138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.517306 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.517221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.517306 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.517258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.517306 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.517297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qw4l\" (UniqueName: \"kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.517450 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.517331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.588386 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.588362    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 19:00:53.617774 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.617742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.617967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.617781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.617967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.617820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qw4l\" (UniqueName: \"kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.617967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.617867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.617967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.617925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.618323 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.618305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.618590 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.618569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.618659 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.618569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.620409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.620388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.627658 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.627632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qw4l\" (UniqueName: \"kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-pmkch\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.718331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.718243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls\") pod \"44f69884-7f94-4674-82bf-7b70f5a955d9\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") "
Sep 16 19:00:53.718331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.718284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config\") pod \"44f69884-7f94-4674-82bf-7b70f5a955d9\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") "
Sep 16 19:00:53.718554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.718334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmg6x\" (UniqueName: \"kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x\") pod \"44f69884-7f94-4674-82bf-7b70f5a955d9\" (UID: \"44f69884-7f94-4674-82bf-7b70f5a955d9\") "
Sep 16 19:00:53.718717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.718687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44f69884-7f94-4674-82bf-7b70f5a955d9" (UID: "44f69884-7f94-4674-82bf-7b70f5a955d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:53.720532 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.720508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44f69884-7f94-4674-82bf-7b70f5a955d9" (UID: "44f69884-7f94-4674-82bf-7b70f5a955d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:53.720532 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.720519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x" (OuterVolumeSpecName: "kube-api-access-hmg6x") pod "44f69884-7f94-4674-82bf-7b70f5a955d9" (UID: "44f69884-7f94-4674-82bf-7b70f5a955d9"). InnerVolumeSpecName "kube-api-access-hmg6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:53.775496 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.775467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:00:53.819218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.819192    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f69884-7f94-4674-82bf-7b70f5a955d9-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:00:53.819218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.819217    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:00:53.819389 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:53.819228    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmg6x\" (UniqueName: \"kubernetes.io/projected/44f69884-7f94-4674-82bf-7b70f5a955d9-kube-api-access-hmg6x\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:00:54.049785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.049760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:00:54.051128 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:00:54.051107    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3adad9_1368_487a_a834_387314ee5ae2.slice/crio-9231907f50b73f6050b8434c9fc7f8c1e52ce630e6f463bd579c5289b9918652 WatchSource:0}: Error finding container 9231907f50b73f6050b8434c9fc7f8c1e52ce630e6f463bd579c5289b9918652: Status 404 returned error can't find the container with id 9231907f50b73f6050b8434c9fc7f8c1e52ce630e6f463bd579c5289b9918652
Sep 16 19:00:54.513460 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.513422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerStarted","Data":"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"}
Sep 16 19:00:54.513923 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.513491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerStarted","Data":"9231907f50b73f6050b8434c9fc7f8c1e52ce630e6f463bd579c5289b9918652"}
Sep 16 19:00:54.515295 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.515274    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"
Sep 16 19:00:54.515395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.515300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5" event={"ID":"44f69884-7f94-4674-82bf-7b70f5a955d9","Type":"ContainerDied","Data":"e08cf5da2eebd59d271b6a3c70b1f9f00871c8aa787ab0357a4d9908601599a7"}
Sep 16 19:00:54.515395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.515360    2578 scope.go:117] "RemoveContainer" containerID="8dff69e25bb68832b1b3dcd10f3986de5c9faed32f5d759bb73778a5788dc593"
Sep 16 19:00:54.524199 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.524177    2578 scope.go:117] "RemoveContainer" containerID="ac54ecb6df726ce7089de836399e9a868f8a0ca217824c496ffdd6f40efc44ae"
Sep 16 19:00:54.547443 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.547416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 19:00:54.551517 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:54.551493    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-7kkj5"]
Sep 16 19:00:55.603277 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:55.603248    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" path="/var/lib/kubelet/pods/44f69884-7f94-4674-82bf-7b70f5a955d9/volumes"
Sep 16 19:00:57.295306 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:00:57.295262    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:01:07.294325 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:07.294288    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:01:09.598501 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:09.598462    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:01:19.602758 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:19.602720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:01:23.528494 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:23.528444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 19:01:23.531160 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:23.529165    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" containerID="cri-o://a21cbebde8d284affd7e4fec70267eeede4acf5c5da7e1dd8052fd1164171af1" gracePeriod=30
Sep 16 19:01:23.531160 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:23.529212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kube-rbac-proxy" containerID="cri-o://9056f4c98d74ee18ec430b3ce1107ae90c08ca364bf5748779efe753f25b571f" gracePeriod=30
Sep 16 19:01:24.642560 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:24.642527    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerID="9056f4c98d74ee18ec430b3ce1107ae90c08ca364bf5748779efe753f25b571f" exitCode=2
Sep 16 19:01:24.643099 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:24.642593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerDied","Data":"9056f4c98d74ee18ec430b3ce1107ae90c08ca364bf5748779efe753f25b571f"}
Sep 16 19:01:24.644280 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:24.644259    2578 generic.go:356] "Generic (PLEG): container finished" podID="df3adad9-1368-487a-a834-387314ee5ae2" containerID="c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e" exitCode=0
Sep 16 19:01:24.644368 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:24.644288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerDied","Data":"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"}
Sep 16 19:01:27.290251 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:27.290206    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 16 19:01:29.598533 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.598488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 19:01:29.674834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.674797    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerID="a21cbebde8d284affd7e4fec70267eeede4acf5c5da7e1dd8052fd1164171af1" exitCode=0
Sep 16 19:01:29.675018 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.674871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerDied","Data":"a21cbebde8d284affd7e4fec70267eeede4acf5c5da7e1dd8052fd1164171af1"}
Sep 16 19:01:29.801937 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.801906    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:01:29.828188 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") pod \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") "
Sep 16 19:01:29.828358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md6ng\" (UniqueName: \"kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng\") pod \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") "
Sep 16 19:01:29.828358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location\") pod \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") "
Sep 16 19:01:29.828358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config\") pod \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") "
Sep 16 19:01:29.828664 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert\") pod \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\" (UID: \"a6f7a3e1-4b83-4ba2-97e2-135f230f1768\") "
Sep 16 19:01:29.828985 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6f7a3e1-4b83-4ba2-97e2-135f230f1768" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:29.828985 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.828980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6f7a3e1-4b83-4ba2-97e2-135f230f1768" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:29.829257 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.829211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6f7a3e1-4b83-4ba2-97e2-135f230f1768" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:29.831162 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.831126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6f7a3e1-4b83-4ba2-97e2-135f230f1768" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:29.831326 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.831289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng" (OuterVolumeSpecName: "kube-api-access-md6ng") pod "a6f7a3e1-4b83-4ba2-97e2-135f230f1768" (UID: "a6f7a3e1-4b83-4ba2-97e2-135f230f1768"). InnerVolumeSpecName "kube-api-access-md6ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:29.929145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.929064    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.929145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.929103    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.929145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.929120    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.929145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.929136    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.929454 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:29.929150    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6ng\" (UniqueName: \"kubernetes.io/projected/a6f7a3e1-4b83-4ba2-97e2-135f230f1768-kube-api-access-md6ng\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:01:30.681417 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.681385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"
Sep 16 19:01:30.681417 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.681401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8" event={"ID":"a6f7a3e1-4b83-4ba2-97e2-135f230f1768","Type":"ContainerDied","Data":"4979c8e85694fbf883ea87504fb258ee1ac6acf768d18c0cebe47eb7625af1d8"}
Sep 16 19:01:30.681955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.681472    2578 scope.go:117] "RemoveContainer" containerID="9056f4c98d74ee18ec430b3ce1107ae90c08ca364bf5748779efe753f25b571f"
Sep 16 19:01:30.691600 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.691574    2578 scope.go:117] "RemoveContainer" containerID="a21cbebde8d284affd7e4fec70267eeede4acf5c5da7e1dd8052fd1164171af1"
Sep 16 19:01:30.700933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.700909    2578 scope.go:117] "RemoveContainer" containerID="2cf5fe8228f7b7dea17a93541cb404220533c97e115bc1e29e0d2571defa21ab"
Sep 16 19:01:30.707041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.707018    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 19:01:30.711948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:30.711920    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-g84x8"]
Sep 16 19:01:31.604516 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:01:31.604485    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" path="/var/lib/kubelet/pods/a6f7a3e1-4b83-4ba2-97e2-135f230f1768/volumes"
Sep 16 19:03:16.180495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:16.180414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerStarted","Data":"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"}
Sep 16 19:03:16.180495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:16.180470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerStarted","Data":"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"}
Sep 16 19:03:16.180495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:16.180494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:16.201042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:16.200992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podStartSLOduration=2.517817973 podStartE2EDuration="2m23.200972919s" totalImagesPullingTime="1m51.213754859s" totalInitContainerRuntime="29.469400087s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:16.200287648 +0000 UTC m=+1055.171764913" watchObservedRunningTime="2026-09-16 19:03:16.200972919 +0000 UTC m=+1055.172450169"
Sep 16 19:03:17.187913 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:17.187881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:17.189343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:17.189314    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 19:03:18.191720 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:18.191673    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 19:03:23.195933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:23.195901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:23.196521 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:23.196494    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 16 19:03:33.197760 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:33.197685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:39.611619 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:39.611590    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:03:39.612012 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:39.611866    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container" containerID="cri-o://0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac" gracePeriod=30
Sep 16 19:03:39.612012 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:39.611902    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kube-rbac-proxy" containerID="cri-o://a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5" gracePeriod=30
Sep 16 19:03:40.292194 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:40.292159    2578 generic.go:356] "Generic (PLEG): container finished" podID="df3adad9-1368-487a-a834-387314ee5ae2" containerID="a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5" exitCode=2
Sep 16 19:03:40.292362 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:40.292215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerDied","Data":"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"}
Sep 16 19:03:41.083594 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.083531    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:41.172292 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location\") pod \"df3adad9-1368-487a-a834-387314ee5ae2\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") "
Sep 16 19:03:41.172477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls\") pod \"df3adad9-1368-487a-a834-387314ee5ae2\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") "
Sep 16 19:03:41.172477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert\") pod \"df3adad9-1368-487a-a834-387314ee5ae2\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") "
Sep 16 19:03:41.172477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config\") pod \"df3adad9-1368-487a-a834-387314ee5ae2\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") "
Sep 16 19:03:41.172477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qw4l\" (UniqueName: \"kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l\") pod \"df3adad9-1368-487a-a834-387314ee5ae2\" (UID: \"df3adad9-1368-487a-a834-387314ee5ae2\") "
Sep 16 19:03:41.172780 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172746    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df3adad9-1368-487a-a834-387314ee5ae2" (UID: "df3adad9-1368-487a-a834-387314ee5ae2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:41.172780 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.172753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df3adad9-1368-487a-a834-387314ee5ae2" (UID: "df3adad9-1368-487a-a834-387314ee5ae2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:41.174659 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.174633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df3adad9-1368-487a-a834-387314ee5ae2" (UID: "df3adad9-1368-487a-a834-387314ee5ae2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:41.174659 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.174646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l" (OuterVolumeSpecName: "kube-api-access-8qw4l") pod "df3adad9-1368-487a-a834-387314ee5ae2" (UID: "df3adad9-1368-487a-a834-387314ee5ae2"). InnerVolumeSpecName "kube-api-access-8qw4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:41.176891 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.176870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df3adad9-1368-487a-a834-387314ee5ae2" (UID: "df3adad9-1368-487a-a834-387314ee5ae2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:41.273406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.273362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qw4l\" (UniqueName: \"kubernetes.io/projected/df3adad9-1368-487a-a834-387314ee5ae2-kube-api-access-8qw4l\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.273406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.273400    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3adad9-1368-487a-a834-387314ee5ae2-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.273406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.273414    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df3adad9-1368-487a-a834-387314ee5ae2-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.273618 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.273426    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.273618 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.273437    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df3adad9-1368-487a-a834-387314ee5ae2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.297755 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.297726    2578 generic.go:356] "Generic (PLEG): container finished" podID="df3adad9-1368-487a-a834-387314ee5ae2" containerID="0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac" exitCode=0
Sep 16 19:03:41.297886 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.297788    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"
Sep 16 19:03:41.297886 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.297788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerDied","Data":"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"}
Sep 16 19:03:41.298082 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.297923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch" event={"ID":"df3adad9-1368-487a-a834-387314ee5ae2","Type":"ContainerDied","Data":"9231907f50b73f6050b8434c9fc7f8c1e52ce630e6f463bd579c5289b9918652"}
Sep 16 19:03:41.298082 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.297951    2578 scope.go:117] "RemoveContainer" containerID="a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"
Sep 16 19:03:41.306813 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.306789    2578 scope.go:117] "RemoveContainer" containerID="0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"
Sep 16 19:03:41.314173 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.314156    2578 scope.go:117] "RemoveContainer" containerID="c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"
Sep 16 19:03:41.321027 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321010    2578 scope.go:117] "RemoveContainer" containerID="a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"
Sep 16 19:03:41.321316 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:03:41.321297    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5\": container with ID starting with a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5 not found: ID does not exist" containerID="a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"
Sep 16 19:03:41.321366 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321326    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5"} err="failed to get container status \"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5\": rpc error: code = NotFound desc = could not find container \"a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5\": container with ID starting with a337166405988e9c12b8299d655a4fdf1091603a49bb3e4c2eda838e7fb8a2a5 not found: ID does not exist"
Sep 16 19:03:41.321366 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321342    2578 scope.go:117] "RemoveContainer" containerID="0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"
Sep 16 19:03:41.321562 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:03:41.321542    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac\": container with ID starting with 0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac not found: ID does not exist" containerID="0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"
Sep 16 19:03:41.321623 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321572    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac"} err="failed to get container status \"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac\": rpc error: code = NotFound desc = could not find container \"0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac\": container with ID starting with 0dfb329fd8040f5bb65cb2e9205b0b049adce696e04c3415b9ba4eace2a116ac not found: ID does not exist"
Sep 16 19:03:41.321623 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321592    2578 scope.go:117] "RemoveContainer" containerID="c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"
Sep 16 19:03:41.321839 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:03:41.321817    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e\": container with ID starting with c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e not found: ID does not exist" containerID="c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"
Sep 16 19:03:41.321920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.321844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e"} err="failed to get container status \"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e\": rpc error: code = NotFound desc = could not find container \"c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e\": container with ID starting with c31c17ecbe992a5f867e26b3505bb955a4726b48d265ad23f409d8792f71f81e not found: ID does not exist"
Sep 16 19:03:41.325759 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.325739    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:03:41.327079 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.327061    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-pmkch"]
Sep 16 19:03:41.602354 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:41.602325    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3adad9-1368-487a-a834-387314ee5ae2" path="/var/lib/kubelet/pods/df3adad9-1368-487a-a834-387314ee5ae2/volumes"
Sep 16 19:03:44.801661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801958    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801969    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801980    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="storage-initializer"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801985    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="storage-initializer"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801992    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.801997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802008    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802013    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802022    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802028    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802034    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="storage-initializer"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802038    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="storage-initializer"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802049    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802055    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802065    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802106    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802114    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3adad9-1368-487a-a834-387314ee5ae2" containerName="kserve-container"
Sep 16 19:03:44.802119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kserve-container"
Sep 16 19:03:44.802694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802132    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f69884-7f94-4674-82bf-7b70f5a955d9" containerName="kube-rbac-proxy"
Sep 16 19:03:44.802694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.802137    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f7a3e1-4b83-4ba2-97e2-135f230f1768" containerName="kserve-container"
Sep 16 19:03:44.832768 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.832739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:03:44.832926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.832876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:44.836002 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.835981    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:03:44.836191 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.835992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:44.836191 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.836040    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:44.836331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.836213    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 16 19:03:44.837235 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.837216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:44.837363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.837292    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:44.837363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:44.837300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:03:45.001663 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.001635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.001818 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.001673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.001818 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.001704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.001818 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.001740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgld\" (UniqueName: \"kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.001818 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.001755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.102671 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.102641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.102821 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.102679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.102821 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.102718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.102821 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.102739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgld\" (UniqueName: \"kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.102821 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.102754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.103157 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.103138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.103428 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.103407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.103469 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.103417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.105321 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.105305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.113142 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.113121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgld\" (UniqueName: \"kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-glmtd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.143124 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.143106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:45.439736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:45.439713    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:03:45.442125 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:03:45.442100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4216d2_9cda_49e8_a95a_54e38d0435fd.slice/crio-42aadd451e16313b5792116d5f950ce5c7b04a39d07035a2dcd68a64448afde9 WatchSource:0}: Error finding container 42aadd451e16313b5792116d5f950ce5c7b04a39d07035a2dcd68a64448afde9: Status 404 returned error can't find the container with id 42aadd451e16313b5792116d5f950ce5c7b04a39d07035a2dcd68a64448afde9
Sep 16 19:03:46.321581 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:46.321545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerStarted","Data":"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"}
Sep 16 19:03:46.322001 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:46.321602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerStarted","Data":"42aadd451e16313b5792116d5f950ce5c7b04a39d07035a2dcd68a64448afde9"}
Sep 16 19:03:51.345165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:51.345128    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerID="a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51" exitCode=0
Sep 16 19:03:51.345678 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:51.345208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerDied","Data":"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"}
Sep 16 19:03:52.351778 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:52.351745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerStarted","Data":"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"}
Sep 16 19:03:52.352203 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:52.351803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerStarted","Data":"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"}
Sep 16 19:03:52.352203 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:52.352027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:52.376657 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:52.376603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" podStartSLOduration=3.259144072 podStartE2EDuration="8.376587121s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.117443049s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:52.37473078 +0000 UTC m=+1091.346208021" watchObservedRunningTime="2026-09-16 19:03:52.376587121 +0000 UTC m=+1091.348064370"
Sep 16 19:03:53.355756 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:53.355720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:03:59.365681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:03:59.365647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:05:21.603841 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:21.603813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:05:31.184027 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.183980    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:05:31.184507 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.184319    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kserve-container" containerID="cri-o://a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092" gracePeriod=30
Sep 16 19:05:31.184507 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.184355    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kube-rbac-proxy" containerID="cri-o://74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b" gracePeriod=30
Sep 16 19:05:31.600416 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.600375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.29:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 19:05:31.753429 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.753399    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerID="74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b" exitCode=2
Sep 16 19:05:31.753603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:31.753478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerDied","Data":"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"}
Sep 16 19:05:32.622607 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.622582    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:05:32.756627 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config\") pod \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") "
Sep 16 19:05:32.756627 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert\") pod \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") "
Sep 16 19:05:32.756627 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location\") pod \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") "
Sep 16 19:05:32.756937 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kgld\" (UniqueName: \"kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld\") pod \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") "
Sep 16 19:05:32.756937 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls\") pod \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\" (UID: \"fc4216d2-9cda-49e8-a95a-54e38d0435fd\") "
Sep 16 19:05:32.757047 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.756973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc4216d2-9cda-49e8-a95a-54e38d0435fd" (UID: "fc4216d2-9cda-49e8-a95a-54e38d0435fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:32.757047 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.757011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc4216d2-9cda-49e8-a95a-54e38d0435fd" (UID: "fc4216d2-9cda-49e8-a95a-54e38d0435fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:32.759301 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.759265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld" (OuterVolumeSpecName: "kube-api-access-5kgld") pod "fc4216d2-9cda-49e8-a95a-54e38d0435fd" (UID: "fc4216d2-9cda-49e8-a95a-54e38d0435fd"). InnerVolumeSpecName "kube-api-access-5kgld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:32.759419 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.759327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc4216d2-9cda-49e8-a95a-54e38d0435fd" (UID: "fc4216d2-9cda-49e8-a95a-54e38d0435fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:32.760674 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760616    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerID="a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092" exitCode=0
Sep 16 19:05:32.760794 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760695    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"
Sep 16 19:05:32.760794 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerDied","Data":"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"}
Sep 16 19:05:32.760932 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd" event={"ID":"fc4216d2-9cda-49e8-a95a-54e38d0435fd","Type":"ContainerDied","Data":"42aadd451e16313b5792116d5f950ce5c7b04a39d07035a2dcd68a64448afde9"}
Sep 16 19:05:32.760932 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760830    2578 scope.go:117] "RemoveContainer" containerID="74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"
Sep 16 19:05:32.761037 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.760987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc4216d2-9cda-49e8-a95a-54e38d0435fd" (UID: "fc4216d2-9cda-49e8-a95a-54e38d0435fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:32.773172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.773155    2578 scope.go:117] "RemoveContainer" containerID="a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"
Sep 16 19:05:32.780769 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.780754    2578 scope.go:117] "RemoveContainer" containerID="a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"
Sep 16 19:05:32.788437 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.788416    2578 scope.go:117] "RemoveContainer" containerID="74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"
Sep 16 19:05:32.788689 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:05:32.788664    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b\": container with ID starting with 74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b not found: ID does not exist" containerID="74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"
Sep 16 19:05:32.788759 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.788697    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b"} err="failed to get container status \"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b\": rpc error: code = NotFound desc = could not find container \"74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b\": container with ID starting with 74230c6af3426a1051a02da08c4e813921fdda324108fc6c6575495d2b10ac5b not found: ID does not exist"
Sep 16 19:05:32.788759 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.788714    2578 scope.go:117] "RemoveContainer" containerID="a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"
Sep 16 19:05:32.788967 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:05:32.788951    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092\": container with ID starting with a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092 not found: ID does not exist" containerID="a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"
Sep 16 19:05:32.789040 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.788971    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092"} err="failed to get container status \"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092\": rpc error: code = NotFound desc = could not find container \"a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092\": container with ID starting with a80235b268a59ba6f2e4eb1a61f551687e5e8a31c28d3fa3aee30aceaf9ec092 not found: ID does not exist"
Sep 16 19:05:32.789040 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.788984    2578 scope.go:117] "RemoveContainer" containerID="a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"
Sep 16 19:05:32.789255 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:05:32.789230    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51\": container with ID starting with a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51 not found: ID does not exist" containerID="a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"
Sep 16 19:05:32.789312 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.789265    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51"} err="failed to get container status \"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51\": rpc error: code = NotFound desc = could not find container \"a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51\": container with ID starting with a79ad73a1d4849182044a16f140f21058ae538069566fc34dde43cd72ec3cf51 not found: ID does not exist"
Sep 16 19:05:32.857882 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.857824    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.857882 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.857885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kgld\" (UniqueName: \"kubernetes.io/projected/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-api-access-5kgld\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.858073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.857923    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4216d2-9cda-49e8-a95a-54e38d0435fd-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.858073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.857938    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.858073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:32.857952    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc4216d2-9cda-49e8-a95a-54e38d0435fd-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:05:33.087442 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:33.087406    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:05:33.093220 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:33.093196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-glmtd"]
Sep 16 19:05:33.604756 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:33.604723    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" path="/var/lib/kubelet/pods/fc4216d2-9cda-49e8-a95a-54e38d0435fd/volumes"
Sep 16 19:05:36.376393 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:05:36.376803 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376714    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kserve-container"
Sep 16 19:05:36.376803 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kserve-container"
Sep 16 19:05:36.376915 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376804    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kube-rbac-proxy"
Sep 16 19:05:36.376915 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kube-rbac-proxy"
Sep 16 19:05:36.376915 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376831    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="storage-initializer"
Sep 16 19:05:36.376915 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376841    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="storage-initializer"
Sep 16 19:05:36.377051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376959    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kserve-container"
Sep 16 19:05:36.377051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.376976    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4216d2-9cda-49e8-a95a-54e38d0435fd" containerName="kube-rbac-proxy"
Sep 16 19:05:36.380271 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.380255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.383398 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.383375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:36.383484 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.383455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:36.383664 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.383651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:05:36.383702 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.383659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 16 19:05:36.384698 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.384679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 16 19:05:36.384794 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.384744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:36.384868 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.384797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:36.393709 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.393685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:05:36.491367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.491320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96qk\" (UniqueName: \"kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.491367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.491371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.491617 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.491456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.491617 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.491516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.491617 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.491542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592499 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592669 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592669 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592669 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96qk\" (UniqueName: \"kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592883 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.592883 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:05:36.592845    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:05:36.592997 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:05:36.592939    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls podName:95f52983-9299-4bfe-aa8e-9b605e77f94f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:37.092920317 +0000 UTC m=+1196.064397558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:05:36.593065 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.592989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.593187 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.593170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.593224 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.593184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:36.602338 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:36.602313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96qk\" (UniqueName: \"kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:37.097249 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.097193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:37.099740 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.099717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:37.291066 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.291025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:05:37.568627 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.568557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:05:37.571951 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:05:37.571922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f52983_9299_4bfe_aa8e_9b605e77f94f.slice/crio-165656f94c0dabfbc1f04363d7ce8969b5e5c13b3f2845fb41a3dd57e490e0b2 WatchSource:0}: Error finding container 165656f94c0dabfbc1f04363d7ce8969b5e5c13b3f2845fb41a3dd57e490e0b2: Status 404 returned error can't find the container with id 165656f94c0dabfbc1f04363d7ce8969b5e5c13b3f2845fb41a3dd57e490e0b2
Sep 16 19:05:37.784338 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.784295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerStarted","Data":"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"}
Sep 16 19:05:37.784503 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:05:37.784429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerStarted","Data":"165656f94c0dabfbc1f04363d7ce8969b5e5c13b3f2845fb41a3dd57e490e0b2"}
Sep 16 19:06:06.901651 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:06.901615    2578 generic.go:356] "Generic (PLEG): container finished" podID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerID="6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4" exitCode=0
Sep 16 19:06:06.902085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:06.901660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerDied","Data":"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"}
Sep 16 19:06:07.909085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:07.909040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerStarted","Data":"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"}
Sep 16 19:06:07.909517 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:07.909102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerStarted","Data":"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"}
Sep 16 19:06:07.909517 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:07.909274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:06:07.932112 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:07.932065    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" podStartSLOduration=2.784724793 podStartE2EDuration="31.932047885s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.147323092s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:07.930294154 +0000 UTC m=+1226.901771404" watchObservedRunningTime="2026-09-16 19:06:07.932047885 +0000 UTC m=+1226.903525134"
Sep 16 19:06:08.914237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:08.914190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:06:14.924949 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:06:14.924919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:07:42.602005 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:42.601973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:07:53.203182 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:53.203147    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:07:53.203764 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:53.203453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kserve-container" containerID="cri-o://3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1" gracePeriod=30
Sep 16 19:07:53.203764 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:53.203497    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kube-rbac-proxy" containerID="cri-o://8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828" gracePeriod=30
Sep 16 19:07:53.344612 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:53.344580    2578 generic.go:356] "Generic (PLEG): container finished" podID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerID="8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828" exitCode=2
Sep 16 19:07:53.344749 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:53.344656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerDied","Data":"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"}
Sep 16 19:07:54.643386 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.643361    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:07:54.739110 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m96qk\" (UniqueName: \"kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk\") pod \"95f52983-9299-4bfe-aa8e-9b605e77f94f\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") "
Sep 16 19:07:54.739110 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config\") pod \"95f52983-9299-4bfe-aa8e-9b605e77f94f\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") "
Sep 16 19:07:54.739110 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") pod \"95f52983-9299-4bfe-aa8e-9b605e77f94f\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") "
Sep 16 19:07:54.739395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert\") pod \"95f52983-9299-4bfe-aa8e-9b605e77f94f\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") "
Sep 16 19:07:54.739395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location\") pod \"95f52983-9299-4bfe-aa8e-9b605e77f94f\" (UID: \"95f52983-9299-4bfe-aa8e-9b605e77f94f\") "
Sep 16 19:07:54.739509 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95f52983-9299-4bfe-aa8e-9b605e77f94f" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:54.739610 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.739583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95f52983-9299-4bfe-aa8e-9b605e77f94f" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:54.741369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.741338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95f52983-9299-4bfe-aa8e-9b605e77f94f" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:54.741471 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.741381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk" (OuterVolumeSpecName: "kube-api-access-m96qk") pod "95f52983-9299-4bfe-aa8e-9b605e77f94f" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f"). InnerVolumeSpecName "kube-api-access-m96qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:54.743592 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.743566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95f52983-9299-4bfe-aa8e-9b605e77f94f" (UID: "95f52983-9299-4bfe-aa8e-9b605e77f94f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:54.839955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.839910    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96qk\" (UniqueName: \"kubernetes.io/projected/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-api-access-m96qk\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:07:54.839955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.839951    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:07:54.839955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.839962    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f52983-9299-4bfe-aa8e-9b605e77f94f-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:07:54.840176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.839973    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f52983-9299-4bfe-aa8e-9b605e77f94f-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:07:54.840176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:54.839983    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f52983-9299-4bfe-aa8e-9b605e77f94f-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:07:55.355020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.354984    2578 generic.go:356] "Generic (PLEG): container finished" podID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerID="3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1" exitCode=0
Sep 16 19:07:55.355020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.355023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerDied","Data":"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"}
Sep 16 19:07:55.355221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.355043    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"
Sep 16 19:07:55.355221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.355068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6" event={"ID":"95f52983-9299-4bfe-aa8e-9b605e77f94f","Type":"ContainerDied","Data":"165656f94c0dabfbc1f04363d7ce8969b5e5c13b3f2845fb41a3dd57e490e0b2"}
Sep 16 19:07:55.355221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.355094    2578 scope.go:117] "RemoveContainer" containerID="8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"
Sep 16 19:07:55.363764 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.363746    2578 scope.go:117] "RemoveContainer" containerID="3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"
Sep 16 19:07:55.371139 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.371123    2578 scope.go:117] "RemoveContainer" containerID="6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"
Sep 16 19:07:55.377357 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.377336    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:07:55.378433 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.378419    2578 scope.go:117] "RemoveContainer" containerID="8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"
Sep 16 19:07:55.378701 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:07:55.378683    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828\": container with ID starting with 8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828 not found: ID does not exist" containerID="8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"
Sep 16 19:07:55.378755 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.378708    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828"} err="failed to get container status \"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828\": rpc error: code = NotFound desc = could not find container \"8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828\": container with ID starting with 8bd610630693b7aa46f0df810ef01a195d0063460afa3159c656efb2be571828 not found: ID does not exist"
Sep 16 19:07:55.378755 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.378723    2578 scope.go:117] "RemoveContainer" containerID="3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"
Sep 16 19:07:55.378944 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:07:55.378927    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1\": container with ID starting with 3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1 not found: ID does not exist" containerID="3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"
Sep 16 19:07:55.378992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.378948    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1"} err="failed to get container status \"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1\": rpc error: code = NotFound desc = could not find container \"3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1\": container with ID starting with 3054690019e47655cd309f0a94ff42d7d50341c1b9309f5b81cce9a3483b9ac1 not found: ID does not exist"
Sep 16 19:07:55.378992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.378967    2578 scope.go:117] "RemoveContainer" containerID="6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"
Sep 16 19:07:55.379192 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:07:55.379179    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4\": container with ID starting with 6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4 not found: ID does not exist" containerID="6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"
Sep 16 19:07:55.379234 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.379194    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4"} err="failed to get container status \"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4\": rpc error: code = NotFound desc = could not find container \"6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4\": container with ID starting with 6f636ae67532a6f6942e90b1e406df62538dadd920c7c107744cad7554ce93a4 not found: ID does not exist"
Sep 16 19:07:55.382897 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.382877    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-fntq6"]
Sep 16 19:07:55.603117 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:55.603090    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" path="/var/lib/kubelet/pods/95f52983-9299-4bfe-aa8e-9b605e77f94f/volumes"
Sep 16 19:07:58.405719 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.405684    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406012    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kserve-container"
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406024    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kserve-container"
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kube-rbac-proxy"
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406045    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kube-rbac-proxy"
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="storage-initializer"
Sep 16 19:07:58.406103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="storage-initializer"
Sep 16 19:07:58.406290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kserve-container"
Sep 16 19:07:58.406290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.406126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f52983-9299-4bfe-aa8e-9b605e77f94f" containerName="kube-rbac-proxy"
Sep 16 19:07:58.409420 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.409402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.412547 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.412528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 19:07:58.412547 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.412536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:07:58.412708 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.412536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:07:58.415976 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.415960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:07:58.416113 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.416092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:07:58.416410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.416373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:07:58.416528 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.416506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:07:58.423660 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.423638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:07:58.469245 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.469219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.469410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.469271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.469410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.469312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.469410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.469370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.469534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.469424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk42j\" (UniqueName: \"kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570006 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.569960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk42j\" (UniqueName: \"kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570006 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570201 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:07:58.570180    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 16 19:07:58.570401 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:07:58.570246    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls podName:b922fb60-965b-42fa-ac2b-93b1d6ef5b23 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:59.070226873 +0000 UTC m=+1338.041704101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 16 19:07:58.570512 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570651 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.570757 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.570742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:58.579689 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:58.579666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk42j\" (UniqueName: \"kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:59.073264 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:59.073223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:59.075689 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:59.075666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:59.319525 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:59.319493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:07:59.593725 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:07:59.593702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:07:59.595621 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:07:59.595593    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb922fb60_965b_42fa_ac2b_93b1d6ef5b23.slice/crio-5943bcb44fc1d7b668d665da4bb7fe80a86fa48de71dad04a851e3cc2b72cffe WatchSource:0}: Error finding container 5943bcb44fc1d7b668d665da4bb7fe80a86fa48de71dad04a851e3cc2b72cffe: Status 404 returned error can't find the container with id 5943bcb44fc1d7b668d665da4bb7fe80a86fa48de71dad04a851e3cc2b72cffe
Sep 16 19:08:00.380279 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:00.380236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerStarted","Data":"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"}
Sep 16 19:08:00.380434 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:00.380321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerStarted","Data":"5943bcb44fc1d7b668d665da4bb7fe80a86fa48de71dad04a851e3cc2b72cffe"}
Sep 16 19:08:05.402071 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:05.402037    2578 generic.go:356] "Generic (PLEG): container finished" podID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerID="2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad" exitCode=0
Sep 16 19:08:05.402453 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:05.402108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerDied","Data":"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"}
Sep 16 19:08:06.409045 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:06.409010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerStarted","Data":"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"}
Sep 16 19:08:06.409423 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:06.409073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerStarted","Data":"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"}
Sep 16 19:08:06.409423 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:06.409396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:08:06.409493 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:06.409464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:08:06.432604 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:06.432555    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" podStartSLOduration=3.41569835 podStartE2EDuration="8.432540959s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.016842609s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:06.43192697 +0000 UTC m=+1345.403404220" watchObservedRunningTime="2026-09-16 19:08:06.432540959 +0000 UTC m=+1345.404018208"
Sep 16 19:08:12.420345 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:08:12.420314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:09:39.603209 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:39.603121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:09:50.641042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:50.640986    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:09:54.687445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:54.687413    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:09:54.687829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:54.687652    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kserve-container" containerID="cri-o://4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0" gracePeriod=30
Sep 16 19:09:54.687829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:54.687669    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kube-rbac-proxy" containerID="cri-o://c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516" gracePeriod=30
Sep 16 19:09:54.856175 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:54.856134    2578 generic.go:356] "Generic (PLEG): container finished" podID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerID="c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516" exitCode=2
Sep 16 19:09:54.856369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:54.856215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerDied","Data":"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"}
Sep 16 19:09:56.022694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.022670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:09:56.148820 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.148732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location\") pod \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") "
Sep 16 19:09:56.148820 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.148768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert\") pod \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") "
Sep 16 19:09:56.149013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.148827    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") pod \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") "
Sep 16 19:09:56.149013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.148900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk42j\" (UniqueName: \"kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j\") pod \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") "
Sep 16 19:09:56.149013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.148950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config\") pod \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\" (UID: \"b922fb60-965b-42fa-ac2b-93b1d6ef5b23\") "
Sep 16 19:09:56.149257 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.149232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b922fb60-965b-42fa-ac2b-93b1d6ef5b23" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:56.149430 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.149413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b922fb60-965b-42fa-ac2b-93b1d6ef5b23" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:56.151107 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.151081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b922fb60-965b-42fa-ac2b-93b1d6ef5b23" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:56.151208 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.151176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j" (OuterVolumeSpecName: "kube-api-access-wk42j") pod "b922fb60-965b-42fa-ac2b-93b1d6ef5b23" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23"). InnerVolumeSpecName "kube-api-access-wk42j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:56.153294 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.153265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b922fb60-965b-42fa-ac2b-93b1d6ef5b23" (UID: "b922fb60-965b-42fa-ac2b-93b1d6ef5b23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:56.250184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.250148    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk42j\" (UniqueName: \"kubernetes.io/projected/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-api-access-wk42j\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:09:56.250184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.250178    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:09:56.250184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.250187    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:09:56.250406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.250197    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:09:56.250406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.250206    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b922fb60-965b-42fa-ac2b-93b1d6ef5b23-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:09:56.866607 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.866573    2578 generic.go:356] "Generic (PLEG): container finished" podID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerID="4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0" exitCode=0
Sep 16 19:09:56.866777 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.866628    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"
Sep 16 19:09:56.866777 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.866641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerDied","Data":"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"}
Sep 16 19:09:56.866777 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.866714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6" event={"ID":"b922fb60-965b-42fa-ac2b-93b1d6ef5b23","Type":"ContainerDied","Data":"5943bcb44fc1d7b668d665da4bb7fe80a86fa48de71dad04a851e3cc2b72cffe"}
Sep 16 19:09:56.866777 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.866741    2578 scope.go:117] "RemoveContainer" containerID="c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"
Sep 16 19:09:56.875713 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.875694    2578 scope.go:117] "RemoveContainer" containerID="4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"
Sep 16 19:09:56.883376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.883350    2578 scope.go:117] "RemoveContainer" containerID="2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"
Sep 16 19:09:56.888711 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.888686    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:09:56.891004 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.890988    2578 scope.go:117] "RemoveContainer" containerID="c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"
Sep 16 19:09:56.891298 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:09:56.891277    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516\": container with ID starting with c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516 not found: ID does not exist" containerID="c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"
Sep 16 19:09:56.891358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.891309    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516"} err="failed to get container status \"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516\": rpc error: code = NotFound desc = could not find container \"c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516\": container with ID starting with c590faad6a0fde64496f34a28b6383ed54ffd848793b0e87fa5b55aa625ce516 not found: ID does not exist"
Sep 16 19:09:56.891358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.891327    2578 scope.go:117] "RemoveContainer" containerID="4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"
Sep 16 19:09:56.891616 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:09:56.891590    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0\": container with ID starting with 4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0 not found: ID does not exist" containerID="4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"
Sep 16 19:09:56.891672 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.891620    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0"} err="failed to get container status \"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0\": rpc error: code = NotFound desc = could not find container \"4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0\": container with ID starting with 4d2aa7a447ee174c38c210210fce376625bb9015cc53fb99201e6f981c9696c0 not found: ID does not exist"
Sep 16 19:09:56.891672 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.891658    2578 scope.go:117] "RemoveContainer" containerID="2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"
Sep 16 19:09:56.891978 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:09:56.891952    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad\": container with ID starting with 2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad not found: ID does not exist" containerID="2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"
Sep 16 19:09:56.892045 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.891987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad"} err="failed to get container status \"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad\": rpc error: code = NotFound desc = could not find container \"2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad\": container with ID starting with 2d065f81cbc4f723e1f797909df272e440baec446b39e279b1eaa32abba06dad not found: ID does not exist"
Sep 16 19:09:56.892385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:56.892365    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qfbc6"]
Sep 16 19:09:57.602217 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:57.602187    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" path="/var/lib/kubelet/pods/b922fb60-965b-42fa-ac2b-93b1d6ef5b23/volumes"
Sep 16 19:09:59.873563 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873526    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873813    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="storage-initializer"
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873824    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="storage-initializer"
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kserve-container"
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kserve-container"
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kube-rbac-proxy"
Sep 16 19:09:59.873946 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kube-rbac-proxy"
Sep 16 19:09:59.874176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873967    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kserve-container"
Sep 16 19:09:59.874176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.873973    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b922fb60-965b-42fa-ac2b-93b1d6ef5b23" containerName="kube-rbac-proxy"
Sep 16 19:09:59.879142 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.879120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:09:59.881923 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.881901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 16 19:09:59.882054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.882004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:59.882054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.882004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:09:59.882054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.882004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:59.882268 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.882108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:09:59.883109 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.883091    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:59.884409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.884390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:59.888590 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.888571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:09:59.981575 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.981543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:09:59.981748 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.981585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:09:59.981748 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.981679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:09:59.981748 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.981716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:09:59.981885 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:09:59.981762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxttx\" (UniqueName: \"kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082481 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxttx\" (UniqueName: \"kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.082888 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.082843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.083318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.083297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.083355 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.083297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.085149 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.085118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.092495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.092473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxttx\" (UniqueName: \"kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.189730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.189647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:00.461948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.461904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:10:00.464547 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:10:00.464518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd175353d_4a1f_4eec_9cb7_b2d06debe223.slice/crio-c48cef05b595a9426cbf6bf8da3dc1ebd9bb5799b7b23f3d8806867d811a6197 WatchSource:0}: Error finding container c48cef05b595a9426cbf6bf8da3dc1ebd9bb5799b7b23f3d8806867d811a6197: Status 404 returned error can't find the container with id c48cef05b595a9426cbf6bf8da3dc1ebd9bb5799b7b23f3d8806867d811a6197
Sep 16 19:10:00.885010 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.884975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerStarted","Data":"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"}
Sep 16 19:10:00.885393 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:00.885034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerStarted","Data":"c48cef05b595a9426cbf6bf8da3dc1ebd9bb5799b7b23f3d8806867d811a6197"}
Sep 16 19:10:05.905888 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:05.905827    2578 generic.go:356] "Generic (PLEG): container finished" podID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerID="3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53" exitCode=0
Sep 16 19:10:05.906250 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:05.905902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerDied","Data":"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"}
Sep 16 19:10:06.912838 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.912802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerStarted","Data":"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"}
Sep 16 19:10:06.913336 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.912888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerStarted","Data":"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"}
Sep 16 19:10:06.913336 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.913204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:06.913458 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.913343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:06.914610 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.914588    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:10:06.936674 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:06.936621    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podStartSLOduration=3.284283542 podStartE2EDuration="7.936605373s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.652321831s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:06.935137009 +0000 UTC m=+1465.906614259" watchObservedRunningTime="2026-09-16 19:10:06.936605373 +0000 UTC m=+1465.908082621"
Sep 16 19:10:07.917671 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:07.917629    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:10:12.922104 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:12.922073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:12.922733 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:12.922706    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:10:22.923435 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:22.923403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:41.787222 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:41.787190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:10:41.787675 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:41.787479    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container" containerID="cri-o://2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9" gracePeriod=30
Sep 16 19:10:41.788008 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:41.787818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kube-rbac-proxy" containerID="cri-o://d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad" gracePeriod=30
Sep 16 19:10:42.062410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:42.062315    2578 generic.go:356] "Generic (PLEG): container finished" podID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerID="d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad" exitCode=2
Sep 16 19:10:42.062410 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:42.062394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerDied","Data":"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"}
Sep 16 19:10:42.917929 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:42.917891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:10:42.923184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:42.923160    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:10:43.426521 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.426494    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:43.532229 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert\") pod \"d175353d-4a1f-4eec-9cb7-b2d06debe223\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") "
Sep 16 19:10:43.532229 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls\") pod \"d175353d-4a1f-4eec-9cb7-b2d06debe223\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") "
Sep 16 19:10:43.532463 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location\") pod \"d175353d-4a1f-4eec-9cb7-b2d06debe223\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") "
Sep 16 19:10:43.532463 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config\") pod \"d175353d-4a1f-4eec-9cb7-b2d06debe223\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") "
Sep 16 19:10:43.532463 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532392    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxttx\" (UniqueName: \"kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx\") pod \"d175353d-4a1f-4eec-9cb7-b2d06debe223\" (UID: \"d175353d-4a1f-4eec-9cb7-b2d06debe223\") "
Sep 16 19:10:43.532613 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d175353d-4a1f-4eec-9cb7-b2d06debe223" (UID: "d175353d-4a1f-4eec-9cb7-b2d06debe223"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:43.532613 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d175353d-4a1f-4eec-9cb7-b2d06debe223" (UID: "d175353d-4a1f-4eec-9cb7-b2d06debe223"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:43.532685 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532670    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:10:43.532719 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.532684    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d175353d-4a1f-4eec-9cb7-b2d06debe223-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:10:43.533203 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.533182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d175353d-4a1f-4eec-9cb7-b2d06debe223" (UID: "d175353d-4a1f-4eec-9cb7-b2d06debe223"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:43.534431 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.534407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d175353d-4a1f-4eec-9cb7-b2d06debe223" (UID: "d175353d-4a1f-4eec-9cb7-b2d06debe223"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:43.534533 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.534514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx" (OuterVolumeSpecName: "kube-api-access-rxttx") pod "d175353d-4a1f-4eec-9cb7-b2d06debe223" (UID: "d175353d-4a1f-4eec-9cb7-b2d06debe223"). InnerVolumeSpecName "kube-api-access-rxttx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:43.633624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.633596    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d175353d-4a1f-4eec-9cb7-b2d06debe223-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:10:43.633624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.633618    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d175353d-4a1f-4eec-9cb7-b2d06debe223-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:10:43.633624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:43.633628    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxttx\" (UniqueName: \"kubernetes.io/projected/d175353d-4a1f-4eec-9cb7-b2d06debe223-kube-api-access-rxttx\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.072883 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.072826    2578 generic.go:356] "Generic (PLEG): container finished" podID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerID="2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9" exitCode=0
Sep 16 19:10:44.073369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.072918    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"
Sep 16 19:10:44.073369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.072912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerDied","Data":"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"}
Sep 16 19:10:44.073369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.073021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9" event={"ID":"d175353d-4a1f-4eec-9cb7-b2d06debe223","Type":"ContainerDied","Data":"c48cef05b595a9426cbf6bf8da3dc1ebd9bb5799b7b23f3d8806867d811a6197"}
Sep 16 19:10:44.073369 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.073059    2578 scope.go:117] "RemoveContainer" containerID="d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"
Sep 16 19:10:44.082688 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.082669    2578 scope.go:117] "RemoveContainer" containerID="2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"
Sep 16 19:10:44.090259 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.090240    2578 scope.go:117] "RemoveContainer" containerID="3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"
Sep 16 19:10:44.096629 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.096608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:10:44.099346 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.099285    2578 scope.go:117] "RemoveContainer" containerID="d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"
Sep 16 19:10:44.100087 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:10:44.100066    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad\": container with ID starting with d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad not found: ID does not exist" containerID="d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"
Sep 16 19:10:44.100163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.100095    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad"} err="failed to get container status \"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad\": rpc error: code = NotFound desc = could not find container \"d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad\": container with ID starting with d345ddd3db4e2d81498a069d08ff53202bb44ece715183db4c5b88ced279bbad not found: ID does not exist"
Sep 16 19:10:44.100163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.100111    2578 scope.go:117] "RemoveContainer" containerID="2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"
Sep 16 19:10:44.100405 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:10:44.100386    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9\": container with ID starting with 2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9 not found: ID does not exist" containerID="2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"
Sep 16 19:10:44.100474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.100419    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9"} err="failed to get container status \"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9\": rpc error: code = NotFound desc = could not find container \"2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9\": container with ID starting with 2e3c6d022fdd57bd487cf94014aa3aa3dc0d01d5c8308d7d1e1c3e32a5547ee9 not found: ID does not exist"
Sep 16 19:10:44.100474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.100443    2578 scope.go:117] "RemoveContainer" containerID="3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"
Sep 16 19:10:44.100741 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:10:44.100719    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53\": container with ID starting with 3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53 not found: ID does not exist" containerID="3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"
Sep 16 19:10:44.100837 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.100744    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53"} err="failed to get container status \"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53\": rpc error: code = NotFound desc = could not find container \"3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53\": container with ID starting with 3386fe12cfac10f212317af842c088d3677e5fb211f2e97fe0ad6d13fc4e3d53 not found: ID does not exist"
Sep 16 19:10:44.101526 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:44.101502    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-g2zg9"]
Sep 16 19:10:45.603217 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:45.603185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" path="/var/lib/kubelet/pods/d175353d-4a1f-4eec-9cb7-b2d06debe223/volumes"
Sep 16 19:10:46.982100 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982067    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982391    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="storage-initializer"
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="storage-initializer"
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container"
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982427    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container"
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kube-rbac-proxy"
Sep 16 19:10:46.982461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kube-rbac-proxy"
Sep 16 19:10:46.982645 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982484    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kube-rbac-proxy"
Sep 16 19:10:46.982645 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.982495    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175353d-4a1f-4eec-9cb7-b2d06debe223" containerName="kserve-container"
Sep 16 19:10:46.988204 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.988186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:46.991430 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.991411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 16 19:10:46.991599 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.991582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:10:46.991675 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.991546    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:10:46.991790 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.991775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:10:46.991885 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.991870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:10:46.992834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.992800    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:10:46.992950 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.992816    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:10:46.997149 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:46.997128    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:10:47.061435 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.061400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncddn\" (UniqueName: \"kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.061573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.061443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.061573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.061538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.061573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.061566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.061691 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.061588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.162504 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.162474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.162681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.162509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.162681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.162531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.162681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.162575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncddn\" (UniqueName: \"kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.162681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.162630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.163126 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.163088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.163250 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.163220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.163337 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.163320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.165253 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.165237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.172693 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.172669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncddn\" (UniqueName: \"kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.299489 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.299421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:47.574737 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:47.574720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:10:47.577129 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:10:47.577100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbaf9b1_b742_4f39_8feb_8198038826c2.slice/crio-c3cb88c816d7d46563eb14ac77676b59050bbb483f16902db1a2dd8f525c12f3 WatchSource:0}: Error finding container c3cb88c816d7d46563eb14ac77676b59050bbb483f16902db1a2dd8f525c12f3: Status 404 returned error can't find the container with id c3cb88c816d7d46563eb14ac77676b59050bbb483f16902db1a2dd8f525c12f3
Sep 16 19:10:48.092812 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:48.092775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerStarted","Data":"b584d1f60830583e901510cbc17f93c8c87fd9e055d0091c144cfa7e774721dc"}
Sep 16 19:10:48.093208 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:48.092867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerStarted","Data":"c3cb88c816d7d46563eb14ac77676b59050bbb483f16902db1a2dd8f525c12f3"}
Sep 16 19:10:56.126136 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:56.126106    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerID="b584d1f60830583e901510cbc17f93c8c87fd9e055d0091c144cfa7e774721dc" exitCode=0
Sep 16 19:10:56.126485 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:56.126186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerDied","Data":"b584d1f60830583e901510cbc17f93c8c87fd9e055d0091c144cfa7e774721dc"}
Sep 16 19:10:57.138545 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.138509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerStarted","Data":"932d9e409765331acfaf4a7734fafcd151681d6177720c689c0097a77010cd04"}
Sep 16 19:10:57.138997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.138588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerStarted","Data":"5d70f16b564bac8f5d27b324db47427196b8bbd5617c676079baa2755f102663"}
Sep 16 19:10:57.138997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.138883    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:57.138997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.138937    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:10:57.140330 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.140305    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:10:57.162221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:57.162171    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podStartSLOduration=3.197232185 podStartE2EDuration="11.162156602s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.964924417s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:57.160955889 +0000 UTC m=+1516.132433139" watchObservedRunningTime="2026-09-16 19:10:57.162156602 +0000 UTC m=+1516.133633850"
Sep 16 19:10:58.144060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:10:58.144013    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:11:03.148503 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:03.148421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:11:03.149037 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:03.149009    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:11:13.150016 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:13.149986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:11:21.970675 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:21.970642    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:11:21.971221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:21.970909    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container" containerID="cri-o://5d70f16b564bac8f5d27b324db47427196b8bbd5617c676079baa2755f102663" gracePeriod=30
Sep 16 19:11:21.971221 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:21.970946    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kube-rbac-proxy" containerID="cri-o://932d9e409765331acfaf4a7734fafcd151681d6177720c689c0097a77010cd04" gracePeriod=30
Sep 16 19:11:22.246523 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:22.246432    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerID="932d9e409765331acfaf4a7734fafcd151681d6177720c689c0097a77010cd04" exitCode=2
Sep 16 19:11:22.246523 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:22.246480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerDied","Data":"932d9e409765331acfaf4a7734fafcd151681d6177720c689c0097a77010cd04"}
Sep 16 19:11:23.144825 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.144795    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 19:11:23.149067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.149043    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:11:23.254122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.254085    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerID="5d70f16b564bac8f5d27b324db47427196b8bbd5617c676079baa2755f102663" exitCode=0
Sep 16 19:11:23.254269 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.254228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerDied","Data":"5d70f16b564bac8f5d27b324db47427196b8bbd5617c676079baa2755f102663"}
Sep 16 19:11:23.304097 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.304075    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:11:23.358401 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls\") pod \"1dbaf9b1-b742-4f39-8feb-8198038826c2\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") "
Sep 16 19:11:23.358553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncddn\" (UniqueName: \"kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn\") pod \"1dbaf9b1-b742-4f39-8feb-8198038826c2\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") "
Sep 16 19:11:23.358553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config\") pod \"1dbaf9b1-b742-4f39-8feb-8198038826c2\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") "
Sep 16 19:11:23.358553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location\") pod \"1dbaf9b1-b742-4f39-8feb-8198038826c2\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") "
Sep 16 19:11:23.358553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert\") pod \"1dbaf9b1-b742-4f39-8feb-8198038826c2\" (UID: \"1dbaf9b1-b742-4f39-8feb-8198038826c2\") "
Sep 16 19:11:23.358920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dbaf9b1-b742-4f39-8feb-8198038826c2" (UID: "1dbaf9b1-b742-4f39-8feb-8198038826c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:23.358920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.358906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dbaf9b1-b742-4f39-8feb-8198038826c2" (UID: "1dbaf9b1-b742-4f39-8feb-8198038826c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:23.359812 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.359784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dbaf9b1-b742-4f39-8feb-8198038826c2" (UID: "1dbaf9b1-b742-4f39-8feb-8198038826c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:23.360518 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.360496    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn" (OuterVolumeSpecName: "kube-api-access-ncddn") pod "1dbaf9b1-b742-4f39-8feb-8198038826c2" (UID: "1dbaf9b1-b742-4f39-8feb-8198038826c2"). InnerVolumeSpecName "kube-api-access-ncddn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:23.360672 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.360658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dbaf9b1-b742-4f39-8feb-8198038826c2" (UID: "1dbaf9b1-b742-4f39-8feb-8198038826c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:23.459237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.459148    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbaf9b1-b742-4f39-8feb-8198038826c2-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:11:23.459237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.459188    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncddn\" (UniqueName: \"kubernetes.io/projected/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-api-access-ncddn\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:11:23.459237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.459199    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:11:23.459237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.459208    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbaf9b1-b742-4f39-8feb-8198038826c2-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:11:23.459237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:23.459218    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbaf9b1-b742-4f39-8feb-8198038826c2-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:11:24.260324 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.260287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt" event={"ID":"1dbaf9b1-b742-4f39-8feb-8198038826c2","Type":"ContainerDied","Data":"c3cb88c816d7d46563eb14ac77676b59050bbb483f16902db1a2dd8f525c12f3"}
Sep 16 19:11:24.260717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.260333    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"
Sep 16 19:11:24.260717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.260407    2578 scope.go:117] "RemoveContainer" containerID="932d9e409765331acfaf4a7734fafcd151681d6177720c689c0097a77010cd04"
Sep 16 19:11:24.268560 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.268539    2578 scope.go:117] "RemoveContainer" containerID="5d70f16b564bac8f5d27b324db47427196b8bbd5617c676079baa2755f102663"
Sep 16 19:11:24.275820 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.275800    2578 scope.go:117] "RemoveContainer" containerID="b584d1f60830583e901510cbc17f93c8c87fd9e055d0091c144cfa7e774721dc"
Sep 16 19:11:24.281521 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.281496    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:11:24.287900 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:24.287880    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-98ntt"]
Sep 16 19:11:25.603110 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:25.603078    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" path="/var/lib/kubelet/pods/1dbaf9b1-b742-4f39-8feb-8198038826c2/volumes"
Sep 16 19:11:27.198395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198360    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="storage-initializer"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198666    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="storage-initializer"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198682    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kube-rbac-proxy"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kube-rbac-proxy"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198734    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kserve-container"
Sep 16 19:11:27.198743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.198741    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbaf9b1-b742-4f39-8feb-8198038826c2" containerName="kube-rbac-proxy"
Sep 16 19:11:27.203444 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.203426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.206468 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.206444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:27.206628 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.206531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:11:27.206628 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.206568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:27.206628 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.206594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:11:27.207752 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.207732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:27.207871 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.207743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:27.207944 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.207871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:11:27.214023 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.214002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:11:27.238123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.238094    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:11:27.241309 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.241293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.244170 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.244145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:11:27.253792 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.253772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:11:27.288067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.288192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.288192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.288192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.288373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.288373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gcs\" (UniqueName: \"kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.288373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.288373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmxb\" (UniqueName: \"kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.288373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.288527 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.288394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.388864 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.388819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.388868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gcs\" (UniqueName: \"kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.388902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.388960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmxb\" (UniqueName: \"kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.388982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389296 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389296 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389296 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389296 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389514 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:11:27.389370    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:11:27.389514 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:11:27.389429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls podName:c426d8af-09e5-4ece-b2b6-cc048d3b9b16 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:27.889409617 +0000 UTC m=+1546.860886877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls") pod "isvc-canary-create-predictor-58f6457999-fmvw4" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:11:27.389514 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.389514 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389796 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.389881 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.389806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.390032 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.390015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.390067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.390017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.391523 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.391502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.400025 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.399998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gcs\" (UniqueName: \"kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.401194 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.401173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmxb\" (UniqueName: \"kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb\") pod \"isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.514465 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.514386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:27.787684 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.787659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:11:27.789565 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:11:27.789530    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfc8bad_6f15_4d95_ae36_f9c8b0d6e1f7.slice/crio-43909e07de6210b74debed10ca4ab95531fa831824795cfe39b67cb1fa20d419 WatchSource:0}: Error finding container 43909e07de6210b74debed10ca4ab95531fa831824795cfe39b67cb1fa20d419: Status 404 returned error can't find the container with id 43909e07de6210b74debed10ca4ab95531fa831824795cfe39b67cb1fa20d419
Sep 16 19:11:27.892948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.892912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:27.895375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:27.895352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") pod \"isvc-canary-create-predictor-58f6457999-fmvw4\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:28.151319 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:28.151286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:28.279818 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:28.279777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerStarted","Data":"0460d1ec597cb599d7fea20809584684b0a68628096077ebb5b6d0373b74c504"}
Sep 16 19:11:28.280195 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:28.279844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerStarted","Data":"43909e07de6210b74debed10ca4ab95531fa831824795cfe39b67cb1fa20d419"}
Sep 16 19:11:28.427654 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:28.427393    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:11:28.430126 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:11:28.430097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc426d8af_09e5_4ece_b2b6_cc048d3b9b16.slice/crio-d39799ecbecb35e0de7ba6b3269a43ed306c38505a57b88944efb746a8089737 WatchSource:0}: Error finding container d39799ecbecb35e0de7ba6b3269a43ed306c38505a57b88944efb746a8089737: Status 404 returned error can't find the container with id d39799ecbecb35e0de7ba6b3269a43ed306c38505a57b88944efb746a8089737
Sep 16 19:11:29.285381 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:29.285348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerStarted","Data":"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"}
Sep 16 19:11:29.285760 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:29.285408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerStarted","Data":"d39799ecbecb35e0de7ba6b3269a43ed306c38505a57b88944efb746a8089737"}
Sep 16 19:11:32.299070 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:32.299031    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerID="0460d1ec597cb599d7fea20809584684b0a68628096077ebb5b6d0373b74c504" exitCode=0
Sep 16 19:11:32.299524 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:32.299104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerDied","Data":"0460d1ec597cb599d7fea20809584684b0a68628096077ebb5b6d0373b74c504"}
Sep 16 19:11:32.300331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:32.300192    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:11:33.304947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:33.304912    2578 generic.go:356] "Generic (PLEG): container finished" podID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerID="38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896" exitCode=0
Sep 16 19:11:33.305308 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:33.304984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerDied","Data":"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"}
Sep 16 19:11:49.416612 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.416576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerStarted","Data":"3abd59b3f154357926a664abed62b342d07aaaca42292cda22ff45ca22c2c061"}
Sep 16 19:11:49.417079 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.416673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerStarted","Data":"f8703f43ecdfa26d27702c190aee2612fa70eea659ac5426b1ddfbf747403cde"}
Sep 16 19:11:49.417079 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.416897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:49.418978 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.418955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerStarted","Data":"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"}
Sep 16 19:11:49.419095 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.419001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerStarted","Data":"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"}
Sep 16 19:11:49.419218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.419200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:49.443363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.443314    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podStartSLOduration=1.8207660350000001 podStartE2EDuration="22.443298401s" totalImagesPullingTime="16.702458601s" totalInitContainerRuntime="3.920073765s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:49.441026134 +0000 UTC m=+1568.412503394" watchObservedRunningTime="2026-09-16 19:11:49.443298401 +0000 UTC m=+1568.414775648"
Sep 16 19:11:49.464673 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:49.464619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podStartSLOduration=2.9213462249999997 podStartE2EDuration="22.464601734s" totalImagesPullingTime="15.720058734s" totalInitContainerRuntime="3.823196775s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:49.462786554 +0000 UTC m=+1568.434263813" watchObservedRunningTime="2026-09-16 19:11:49.464601734 +0000 UTC m=+1568.436078983"
Sep 16 19:11:50.423588 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:50.423552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:50.423957 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:50.423615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:50.424605 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:50.424572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:50.424605 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:50.424586    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:11:51.427835 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:51.427791    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:11:51.428328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:51.428001    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:11:56.432522 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:56.432489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:11:56.433112 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:56.433089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:11:56.433165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:56.433131    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:11:56.433653 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:11:56.433632    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:06.433223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:06.433176    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:06.433693 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:06.433592    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:16.433819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:16.433771    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:16.434332 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:16.433771    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:26.433987 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:26.433943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:26.463266 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:26.433943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:36.433239 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:36.433150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:36.433741 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:36.433651    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:46.434039 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:46.433995    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:46.434538 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:46.433995    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:56.433156 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:56.433103    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:12:56.433654 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:56.433560    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:12:58.598509 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:12:58.598459    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:13:06.434471 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:06.434439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:13:08.599906 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:08.599839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:13:08.825717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:08.825681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:13:08.825953 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:08.825924    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container" containerID="cri-o://b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f" gracePeriod=30
Sep 16 19:13:08.826040 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:08.825951    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kube-rbac-proxy" containerID="cri-o://7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef" gracePeriod=30
Sep 16 19:13:09.762534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:09.762495    2578 generic.go:356] "Generic (PLEG): container finished" podID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerID="7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef" exitCode=2
Sep 16 19:13:09.762985 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:09.762569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerDied","Data":"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"}
Sep 16 19:13:11.429034 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:11.428983    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:13:12.304735 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:12.304703    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:13:12.305047 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:12.305016    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container" containerID="cri-o://f8703f43ecdfa26d27702c190aee2612fa70eea659ac5426b1ddfbf747403cde" gracePeriod=30
Sep 16 19:13:12.305165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:12.305089    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kube-rbac-proxy" containerID="cri-o://3abd59b3f154357926a664abed62b342d07aaaca42292cda22ff45ca22c2c061" gracePeriod=30
Sep 16 19:13:12.777308 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:12.777274    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerID="3abd59b3f154357926a664abed62b342d07aaaca42292cda22ff45ca22c2c061" exitCode=2
Sep 16 19:13:12.777705 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:12.777337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerDied","Data":"3abd59b3f154357926a664abed62b342d07aaaca42292cda22ff45ca22c2c061"}
Sep 16 19:13:14.082274 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.082250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:13:14.126155 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location\") pod \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") "
Sep 16 19:13:14.126155 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") pod \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") "
Sep 16 19:13:14.126377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert\") pod \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") "
Sep 16 19:13:14.126377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config\") pod \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") "
Sep 16 19:13:14.126377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4gcs\" (UniqueName: \"kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs\") pod \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\" (UID: \"c426d8af-09e5-4ece-b2b6-cc048d3b9b16\") "
Sep 16 19:13:14.126533 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c426d8af-09e5-4ece-b2b6-cc048d3b9b16" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:14.126533 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126480    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:14.126641 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c426d8af-09e5-4ece-b2b6-cc048d3b9b16" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:14.126641 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.126633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c426d8af-09e5-4ece-b2b6-cc048d3b9b16" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:14.128607 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.128578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs" (OuterVolumeSpecName: "kube-api-access-f4gcs") pod "c426d8af-09e5-4ece-b2b6-cc048d3b9b16" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16"). InnerVolumeSpecName "kube-api-access-f4gcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:14.128730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.128641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c426d8af-09e5-4ece-b2b6-cc048d3b9b16" (UID: "c426d8af-09e5-4ece-b2b6-cc048d3b9b16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:14.226998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.226958    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:14.226998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.226991    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gcs\" (UniqueName: \"kubernetes.io/projected/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-kube-api-access-f4gcs\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:14.226998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.227001    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:14.226998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.227010    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c426d8af-09e5-4ece-b2b6-cc048d3b9b16-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:14.787254 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.787217    2578 generic.go:356] "Generic (PLEG): container finished" podID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerID="b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f" exitCode=0
Sep 16 19:13:14.787441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.787285    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"
Sep 16 19:13:14.787441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.787314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerDied","Data":"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"}
Sep 16 19:13:14.787441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.787379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4" event={"ID":"c426d8af-09e5-4ece-b2b6-cc048d3b9b16","Type":"ContainerDied","Data":"d39799ecbecb35e0de7ba6b3269a43ed306c38505a57b88944efb746a8089737"}
Sep 16 19:13:14.787441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.787408    2578 scope.go:117] "RemoveContainer" containerID="7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"
Sep 16 19:13:14.799056 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.799029    2578 scope.go:117] "RemoveContainer" containerID="b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"
Sep 16 19:13:14.807477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.807455    2578 scope.go:117] "RemoveContainer" containerID="38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"
Sep 16 19:13:14.815695 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.815673    2578 scope.go:117] "RemoveContainer" containerID="7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"
Sep 16 19:13:14.816229 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:13:14.816201    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef\": container with ID starting with 7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef not found: ID does not exist" containerID="7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"
Sep 16 19:13:14.816345 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.816269    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef"} err="failed to get container status \"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef\": rpc error: code = NotFound desc = could not find container \"7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef\": container with ID starting with 7ee8fe2a65068658fd360ca6369aeac8c6249ff6974403390108740882ddaaef not found: ID does not exist"
Sep 16 19:13:14.816345 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.816291    2578 scope.go:117] "RemoveContainer" containerID="b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"
Sep 16 19:13:14.816723 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:13:14.816692    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f\": container with ID starting with b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f not found: ID does not exist" containerID="b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"
Sep 16 19:13:14.816871 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.816769    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f"} err="failed to get container status \"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f\": rpc error: code = NotFound desc = could not find container \"b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f\": container with ID starting with b2a245020a4e9ae99a03c926638c48fd0de8b9487bc241e4746ddc680c19156f not found: ID does not exist"
Sep 16 19:13:14.816871 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.816792    2578 scope.go:117] "RemoveContainer" containerID="38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"
Sep 16 19:13:14.817450 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:13:14.817310    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896\": container with ID starting with 38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896 not found: ID does not exist" containerID="38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"
Sep 16 19:13:14.817450 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.817340    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896"} err="failed to get container status \"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896\": rpc error: code = NotFound desc = could not find container \"38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896\": container with ID starting with 38ceafa0adf1e21d4bf001bb19316a2ff915ece62df1a711a6e9d2047c705896 not found: ID does not exist"
Sep 16 19:13:14.819217 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.819197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:13:14.822938 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:14.822909    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-58f6457999-fmvw4"]
Sep 16 19:13:15.603381 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:15.603344    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" path="/var/lib/kubelet/pods/c426d8af-09e5-4ece-b2b6-cc048d3b9b16/volumes"
Sep 16 19:13:16.428060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:16.428014    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:13:17.516715 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.516677    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517064    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container"
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517076    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container"
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517083    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kube-rbac-proxy"
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517088    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kube-rbac-proxy"
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517096    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="storage-initializer"
Sep 16 19:13:17.517123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517101    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="storage-initializer"
Sep 16 19:13:17.517313 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517175    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kube-rbac-proxy"
Sep 16 19:13:17.517313 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.517184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c426d8af-09e5-4ece-b2b6-cc048d3b9b16" containerName="kserve-container"
Sep 16 19:13:17.520365 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.520344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.523530 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.523505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:13:17.523659 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.523508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:13:17.532589 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.532553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:13:17.556199 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.556155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.556385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.556254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.556385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.556296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.556385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.556325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.556385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.556380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pzc\" (UniqueName: \"kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.657841 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.657801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.657893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.657927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.657953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.657991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pzc\" (UniqueName: \"kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658212 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:13:17.658090    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:13:17.658212 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:13:17.658180    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls podName:285bb0ef-6f8b-4830-95d8-d9bd497d0843 nodeName:}" failed. No retries permitted until 2026-09-16 19:13:18.15815843 +0000 UTC m=+1657.129635657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls") pod "isvc-canary-promote-predictor-5c9955746c-98fpz" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:13:17.658442 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.658414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658608 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.658591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.658651 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.658592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.667736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.667705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pzc\" (UniqueName: \"kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:17.803535 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.803501    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerID="f8703f43ecdfa26d27702c190aee2612fa70eea659ac5426b1ddfbf747403cde" exitCode=0
Sep 16 19:13:17.803731 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.803579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerDied","Data":"f8703f43ecdfa26d27702c190aee2612fa70eea659ac5426b1ddfbf747403cde"}
Sep 16 19:13:17.853382 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.853346    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:13:17.960086 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960033    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls\") pod \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") "
Sep 16 19:13:17.960086 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location\") pod \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") "
Sep 16 19:13:17.960358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jmxb\" (UniqueName: \"kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb\") pod \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") "
Sep 16 19:13:17.960358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config\") pod \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") "
Sep 16 19:13:17.960358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert\") pod \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\" (UID: \"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7\") "
Sep 16 19:13:17.960516 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" (UID: "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.960581 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" (UID: "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:17.960681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.960658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" (UID: "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:17.962393 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.962369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" (UID: "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:17.962493 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:17.962397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb" (OuterVolumeSpecName: "kube-api-access-4jmxb") pod "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" (UID: "9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7"). InnerVolumeSpecName "kube-api-access-4jmxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:18.061230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.061145    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.061230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.061173    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.061230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.061183    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.061230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.061192    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jmxb\" (UniqueName: \"kubernetes.io/projected/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-api-access-4jmxb\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.061230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.061201    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.162381 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.162328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:18.164914 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.164895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") pod \"isvc-canary-promote-predictor-5c9955746c-98fpz\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:18.431395 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.431358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:18.708741 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.708657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:13:18.712083 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:13:18.712052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285bb0ef_6f8b_4830_95d8_d9bd497d0843.slice/crio-b0466d44433c5ba804a1b48d42cbd45a928486f4795b87e64e877e7da591b120 WatchSource:0}: Error finding container b0466d44433c5ba804a1b48d42cbd45a928486f4795b87e64e877e7da591b120: Status 404 returned error can't find the container with id b0466d44433c5ba804a1b48d42cbd45a928486f4795b87e64e877e7da591b120
Sep 16 19:13:18.810021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.809982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerStarted","Data":"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"}
Sep 16 19:13:18.810415 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.810054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerStarted","Data":"b0466d44433c5ba804a1b48d42cbd45a928486f4795b87e64e877e7da591b120"}
Sep 16 19:13:18.811948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.811919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg" event={"ID":"9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7","Type":"ContainerDied","Data":"43909e07de6210b74debed10ca4ab95531fa831824795cfe39b67cb1fa20d419"}
Sep 16 19:13:18.811948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.811951    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"
Sep 16 19:13:18.812131 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.811990    2578 scope.go:117] "RemoveContainer" containerID="3abd59b3f154357926a664abed62b342d07aaaca42292cda22ff45ca22c2c061"
Sep 16 19:13:18.821326 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.821305    2578 scope.go:117] "RemoveContainer" containerID="f8703f43ecdfa26d27702c190aee2612fa70eea659ac5426b1ddfbf747403cde"
Sep 16 19:13:18.829731 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.829707    2578 scope.go:117] "RemoveContainer" containerID="0460d1ec597cb599d7fea20809584684b0a68628096077ebb5b6d0373b74c504"
Sep 16 19:13:18.849256 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.849224    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:13:18.853207 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:18.853178    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6bbbd567bb-gcjvg"]
Sep 16 19:13:19.603752 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:19.603701    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" path="/var/lib/kubelet/pods/9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7/volumes"
Sep 16 19:13:22.831331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:22.831294    2578 generic.go:356] "Generic (PLEG): container finished" podID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerID="74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1" exitCode=0
Sep 16 19:13:22.831754 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:22.831367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerDied","Data":"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"}
Sep 16 19:13:23.838461 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:23.838422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerStarted","Data":"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"}
Sep 16 19:13:23.838889 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:23.838487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerStarted","Data":"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"}
Sep 16 19:13:23.838889 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:23.838677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:23.861976 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:23.861918    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podStartSLOduration=2.9229904380000002 podStartE2EDuration="6.861899139s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.938908701s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:23.860616345 +0000 UTC m=+1662.832093605" watchObservedRunningTime="2026-09-16 19:13:23.861899139 +0000 UTC m=+1662.833376388"
Sep 16 19:13:24.843065 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:24.843027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:24.844434 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:24.844406    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:13:25.848215 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:25.848161    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:13:30.852521 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:30.852489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:13:30.853215 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:30.853184    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:13:40.853211 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:40.853170    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:13:50.853883 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:13:50.853832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:14:00.853792 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:00.853748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:14:10.853604 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:10.853515    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:14:20.853099 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:20.853055    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:14:26.564717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.564682    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="storage-initializer"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565035    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="storage-initializer"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565043    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kube-rbac-proxy"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565050    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kube-rbac-proxy"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565064    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565070    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kserve-container"
Sep 16 19:14:26.565172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.565127    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfc8bad-6f15-4d95-ae36-f9c8b0d6e1f7" containerName="kube-rbac-proxy"
Sep 16 19:14:26.568343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.568327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.571084 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.571064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 19:14:26.571212 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.571070    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 19:14:26.579767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.579743    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:14:26.651001 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.650972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.651174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.651010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.651174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.651043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.651174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.651107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.651174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.651162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmdh\" (UniqueName: \"kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmdh\" (UniqueName: \"kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752469 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752469 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.752783 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.752761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.753106 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.753088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.753163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.753095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.754837 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.754822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.761746 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.761724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmdh\" (UniqueName: \"kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh\") pod \"isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:26.878597 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:26.878568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:27.155899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:27.155835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:14:27.158633 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:14:27.158604    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c38604_f70b_4e26_9446_6d729cdd9297.slice/crio-eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2 WatchSource:0}: Error finding container eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2: Status 404 returned error can't find the container with id eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2
Sep 16 19:14:28.100319 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:28.100275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerStarted","Data":"70dd906d3b279f87fc106441fb0ca415dc14503abd1c442a39696c948ed2e144"}
Sep 16 19:14:28.100760 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:28.100329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerStarted","Data":"eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2"}
Sep 16 19:14:30.853817 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:30.853775    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:14:31.115910 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:31.115802    2578 generic.go:356] "Generic (PLEG): container finished" podID="66c38604-f70b-4e26-9446-6d729cdd9297" containerID="70dd906d3b279f87fc106441fb0ca415dc14503abd1c442a39696c948ed2e144" exitCode=0
Sep 16 19:14:31.115910 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:31.115882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerDied","Data":"70dd906d3b279f87fc106441fb0ca415dc14503abd1c442a39696c948ed2e144"}
Sep 16 19:14:32.122123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:32.122083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerStarted","Data":"796913601f0bb76f4a348b6c5b80dbce9185eae89dcc6e8cce51da30c85abf28"}
Sep 16 19:14:32.122615 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:32.122156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerStarted","Data":"376fab93912e3255fbcfa4989209fc85bd1b4bcd4be88599e30862c5661f9569"}
Sep 16 19:14:32.122615 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:32.122391    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:32.153097 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:32.153045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podStartSLOduration=2.278891162 podStartE2EDuration="6.153029213s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874138051s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:32.15190995 +0000 UTC m=+1731.123387201" watchObservedRunningTime="2026-09-16 19:14:32.153029213 +0000 UTC m=+1731.124506461"
Sep 16 19:14:33.126991 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:33.126956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:33.128536 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:33.128500    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:14:34.131409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:34.131367    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:14:39.136524 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:39.136492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:14:39.137015 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:39.136991    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:14:40.193400 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:40.193365    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:14:40.193968 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:40.193637    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container" containerID="cri-o://cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0" gracePeriod=30
Sep 16 19:14:40.193968 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:40.193718    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kube-rbac-proxy" containerID="cri-o://087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3" gracePeriod=30
Sep 16 19:14:40.194815 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:40.194795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:14:40.849294 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:40.849251    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:14:41.162048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:41.161959    2578 generic.go:356] "Generic (PLEG): container finished" podID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerID="087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3" exitCode=2
Sep 16 19:14:41.162048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:41.162028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerDied","Data":"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"}
Sep 16 19:14:44.938783 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:44.938758    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:14:45.002561 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location\") pod \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") "
Sep 16 19:14:45.002561 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002528    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4pzc\" (UniqueName: \"kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc\") pod \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") "
Sep 16 19:14:45.002774 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert\") pod \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") "
Sep 16 19:14:45.002774 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") pod \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") "
Sep 16 19:14:45.002774 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config\") pod \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\" (UID: \"285bb0ef-6f8b-4830-95d8-d9bd497d0843\") "
Sep 16 19:14:45.002967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "285bb0ef-6f8b-4830-95d8-d9bd497d0843" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:45.003031 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.002994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "285bb0ef-6f8b-4830-95d8-d9bd497d0843" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:45.003073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.003053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "285bb0ef-6f8b-4830-95d8-d9bd497d0843" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:45.004751 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.004730    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "285bb0ef-6f8b-4830-95d8-d9bd497d0843" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:45.004829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.004778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc" (OuterVolumeSpecName: "kube-api-access-b4pzc") pod "285bb0ef-6f8b-4830-95d8-d9bd497d0843" (UID: "285bb0ef-6f8b-4830-95d8-d9bd497d0843"). InnerVolumeSpecName "kube-api-access-b4pzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:45.103385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.103350    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.103385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.103383    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285bb0ef-6f8b-4830-95d8-d9bd497d0843-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.103563 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.103397    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.103563 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.103410    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.103563 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.103422    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pzc\" (UniqueName: \"kubernetes.io/projected/285bb0ef-6f8b-4830-95d8-d9bd497d0843-kube-api-access-b4pzc\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.180133 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.180096    2578 generic.go:356] "Generic (PLEG): container finished" podID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerID="cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0" exitCode=0
Sep 16 19:14:45.180285 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.180170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"
Sep 16 19:14:45.180285 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.180166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerDied","Data":"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"}
Sep 16 19:14:45.180359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.180307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz" event={"ID":"285bb0ef-6f8b-4830-95d8-d9bd497d0843","Type":"ContainerDied","Data":"b0466d44433c5ba804a1b48d42cbd45a928486f4795b87e64e877e7da591b120"}
Sep 16 19:14:45.180359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.180349    2578 scope.go:117] "RemoveContainer" containerID="087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"
Sep 16 19:14:45.189610 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.189593    2578 scope.go:117] "RemoveContainer" containerID="cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"
Sep 16 19:14:45.196978 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.196956    2578 scope.go:117] "RemoveContainer" containerID="74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"
Sep 16 19:14:45.204123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.204106    2578 scope.go:117] "RemoveContainer" containerID="087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"
Sep 16 19:14:45.204422 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:14:45.204399    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3\": container with ID starting with 087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3 not found: ID does not exist" containerID="087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"
Sep 16 19:14:45.204478 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.204432    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3"} err="failed to get container status \"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3\": rpc error: code = NotFound desc = could not find container \"087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3\": container with ID starting with 087ad9826d65239a15f65d83c61b6e8c43d8b386e9752e02a763e1980b3252c3 not found: ID does not exist"
Sep 16 19:14:45.204478 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.204449    2578 scope.go:117] "RemoveContainer" containerID="cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"
Sep 16 19:14:45.204698 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:14:45.204677    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0\": container with ID starting with cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0 not found: ID does not exist" containerID="cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"
Sep 16 19:14:45.204743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.204705    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0"} err="failed to get container status \"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0\": rpc error: code = NotFound desc = could not find container \"cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0\": container with ID starting with cd095c82eb92ed8dc4087676ec1454952f7264cb1907bad785ed43d6472784c0 not found: ID does not exist"
Sep 16 19:14:45.204743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.204722    2578 scope.go:117] "RemoveContainer" containerID="74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"
Sep 16 19:14:45.204998 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:14:45.204976    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1\": container with ID starting with 74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1 not found: ID does not exist" containerID="74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"
Sep 16 19:14:45.205047 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.205005    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1"} err="failed to get container status \"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1\": rpc error: code = NotFound desc = could not find container \"74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1\": container with ID starting with 74da9edede6d243d8219942a4880a788e4d8ffb7d13faa55a93db3ef1138eba1 not found: ID does not exist"
Sep 16 19:14:45.215391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.215370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:14:45.226760 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.226740    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5c9955746c-98fpz"]
Sep 16 19:14:45.603562 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:45.603519    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" path="/var/lib/kubelet/pods/285bb0ef-6f8b-4830-95d8-d9bd497d0843/volumes"
Sep 16 19:14:47.759163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759131    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759450    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kube-rbac-proxy"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kube-rbac-proxy"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759483    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="storage-initializer"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="storage-initializer"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759532    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kube-rbac-proxy"
Sep 16 19:14:47.759543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.759542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="285bb0ef-6f8b-4830-95d8-d9bd497d0843" containerName="kserve-container"
Sep 16 19:14:47.764189 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.764168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.768367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.768346    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:14:47.768467 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.768394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:14:47.778333 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.778313    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:14:47.820680 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.820645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcxg\" (UniqueName: \"kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.820823 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.820689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.820823 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.820717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.820823 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.820743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.820978 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.820825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.921713 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.921675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.921713 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.921717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcxg\" (UniqueName: \"kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.921832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.921888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.921921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922323 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.922297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922450 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.922433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.922495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.922434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.924376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.924355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:47.932137 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:47.932113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcxg\" (UniqueName: \"kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg\") pod \"isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:48.073689 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:48.073602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:48.352094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:48.352066    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:14:48.354746 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:14:48.354718    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece67d8a_3382_414a_b84d_253d94e9b4d7.slice/crio-ff726a951d4494b7337d132596f1088ad95d96b21588f5135e7026c6acfbc556 WatchSource:0}: Error finding container ff726a951d4494b7337d132596f1088ad95d96b21588f5135e7026c6acfbc556: Status 404 returned error can't find the container with id ff726a951d4494b7337d132596f1088ad95d96b21588f5135e7026c6acfbc556
Sep 16 19:14:49.137758 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:49.137710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:14:49.198960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:49.198926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerStarted","Data":"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"}
Sep 16 19:14:49.199123 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:49.198986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerStarted","Data":"ff726a951d4494b7337d132596f1088ad95d96b21588f5135e7026c6acfbc556"}
Sep 16 19:14:53.217794 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:53.217757    2578 generic.go:356] "Generic (PLEG): container finished" podID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerID="d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2" exitCode=0
Sep 16 19:14:53.218281 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:53.217831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerDied","Data":"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"}
Sep 16 19:14:54.225721 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:54.225682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerStarted","Data":"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"}
Sep 16 19:14:54.226174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:54.225748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerStarted","Data":"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"}
Sep 16 19:14:54.226174 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:54.225967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:54.260974 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:54.260920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podStartSLOduration=3.181474268 podStartE2EDuration="7.26090315s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079428882s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:54.260753914 +0000 UTC m=+1753.232231162" watchObservedRunningTime="2026-09-16 19:14:54.26090315 +0000 UTC m=+1753.232380398"
Sep 16 19:14:55.230375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:55.230326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:14:55.231825 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:55.231786    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:14:56.236629 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:56.236589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:14:59.137680 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:14:59.137636    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:15:01.241406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:01.241377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:15:01.242014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:01.241983    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:09.137433 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:09.137377    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:15:11.242062 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:11.242023    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:19.137088 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:19.137043    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:15:21.242214 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:21.242178    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:29.137994 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:29.137956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:15:31.241950 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:31.241911    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:39.137965 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:39.137875    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:15:41.241901 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:41.241838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:49.138038 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:49.138008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:15:51.242821 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:51.242783    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:15:56.672729 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:56.672691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:15:56.673238 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:56.673024    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" containerID="cri-o://376fab93912e3255fbcfa4989209fc85bd1b4bcd4be88599e30862c5661f9569" gracePeriod=30
Sep 16 19:15:56.673238 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:56.673081    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kube-rbac-proxy" containerID="cri-o://796913601f0bb76f4a348b6c5b80dbce9185eae89dcc6e8cce51da30c85abf28" gracePeriod=30
Sep 16 19:15:57.492528 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:57.492495    2578 generic.go:356] "Generic (PLEG): container finished" podID="66c38604-f70b-4e26-9446-6d729cdd9297" containerID="796913601f0bb76f4a348b6c5b80dbce9185eae89dcc6e8cce51da30c85abf28" exitCode=2
Sep 16 19:15:57.492718 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:57.492575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerDied","Data":"796913601f0bb76f4a348b6c5b80dbce9185eae89dcc6e8cce51da30c85abf28"}
Sep 16 19:15:59.131538 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:59.131490    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:15:59.137326 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:15:59.137282    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:16:01.242265 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.242226    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:16:01.512040 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.512012    2578 generic.go:356] "Generic (PLEG): container finished" podID="66c38604-f70b-4e26-9446-6d729cdd9297" containerID="376fab93912e3255fbcfa4989209fc85bd1b4bcd4be88599e30862c5661f9569" exitCode=0
Sep 16 19:16:01.512198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.512053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerDied","Data":"376fab93912e3255fbcfa4989209fc85bd1b4bcd4be88599e30862c5661f9569"}
Sep 16 19:16:01.512198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.512098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q" event={"ID":"66c38604-f70b-4e26-9446-6d729cdd9297","Type":"ContainerDied","Data":"eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2"}
Sep 16 19:16:01.512198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.512122    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eef7911fe354463855b55fecdae09e0b70867a8d01e9ce449e7a5cf00594adf2"
Sep 16 19:16:01.514809 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.514790    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:16:01.623758 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.623730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location\") pod \"66c38604-f70b-4e26-9446-6d729cdd9297\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") "
Sep 16 19:16:01.623758 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.623768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls\") pod \"66c38604-f70b-4e26-9446-6d729cdd9297\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") "
Sep 16 19:16:01.624038 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.623814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert\") pod \"66c38604-f70b-4e26-9446-6d729cdd9297\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") "
Sep 16 19:16:01.624038 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.623873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnmdh\" (UniqueName: \"kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh\") pod \"66c38604-f70b-4e26-9446-6d729cdd9297\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") "
Sep 16 19:16:01.624038 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.623893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config\") pod \"66c38604-f70b-4e26-9446-6d729cdd9297\" (UID: \"66c38604-f70b-4e26-9446-6d729cdd9297\") "
Sep 16 19:16:01.624236 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.624189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66c38604-f70b-4e26-9446-6d729cdd9297" (UID: "66c38604-f70b-4e26-9446-6d729cdd9297"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:01.624308 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.624261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66c38604-f70b-4e26-9446-6d729cdd9297" (UID: "66c38604-f70b-4e26-9446-6d729cdd9297"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:01.624308 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.624257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66c38604-f70b-4e26-9446-6d729cdd9297" (UID: "66c38604-f70b-4e26-9446-6d729cdd9297"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:01.626566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.626545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66c38604-f70b-4e26-9446-6d729cdd9297" (UID: "66c38604-f70b-4e26-9446-6d729cdd9297"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:01.626566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.626553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh" (OuterVolumeSpecName: "kube-api-access-wnmdh") pod "66c38604-f70b-4e26-9446-6d729cdd9297" (UID: "66c38604-f70b-4e26-9446-6d729cdd9297"). InnerVolumeSpecName "kube-api-access-wnmdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:01.725290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.725252    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmdh\" (UniqueName: \"kubernetes.io/projected/66c38604-f70b-4e26-9446-6d729cdd9297-kube-api-access-wnmdh\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:01.725290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.725286    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:01.725290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.725295    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c38604-f70b-4e26-9446-6d729cdd9297-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:01.725569 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.725305    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c38604-f70b-4e26-9446-6d729cdd9297-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:01.725569 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:01.725326    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c38604-f70b-4e26-9446-6d729cdd9297-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.516289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:02.516258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"
Sep 16 19:16:02.540638 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:02.540610    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:16:02.545000 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:02.544976    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-65cc6d8479-ch56q"]
Sep 16 19:16:03.603254 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:03.603218    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" path="/var/lib/kubelet/pods/66c38604-f70b-4e26-9446-6d729cdd9297/volumes"
Sep 16 19:16:11.242604 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:11.242568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:16:17.831918 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:17.831876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:16:17.832428 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:17.832120    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" containerID="cri-o://1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7" gracePeriod=30
Sep 16 19:16:17.832428 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:17.832163    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kube-rbac-proxy" containerID="cri-o://cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e" gracePeriod=30
Sep 16 19:16:18.587310 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:18.587275    2578 generic.go:356] "Generic (PLEG): container finished" podID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerID="cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e" exitCode=2
Sep 16 19:16:18.587474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:18.587320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerDied","Data":"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"}
Sep 16 19:16:21.237445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:21.237405    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 19:16:21.241991 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:21.241965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:16:22.482340 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.482313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:16:22.593462 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkcxg\" (UniqueName: \"kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg\") pod \"ece67d8a-3382-414a-b84d-253d94e9b4d7\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") "
Sep 16 19:16:22.593643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls\") pod \"ece67d8a-3382-414a-b84d-253d94e9b4d7\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") "
Sep 16 19:16:22.593643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert\") pod \"ece67d8a-3382-414a-b84d-253d94e9b4d7\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") "
Sep 16 19:16:22.593643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config\") pod \"ece67d8a-3382-414a-b84d-253d94e9b4d7\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") "
Sep 16 19:16:22.593643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location\") pod \"ece67d8a-3382-414a-b84d-253d94e9b4d7\" (UID: \"ece67d8a-3382-414a-b84d-253d94e9b4d7\") "
Sep 16 19:16:22.593967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ece67d8a-3382-414a-b84d-253d94e9b4d7" (UID: "ece67d8a-3382-414a-b84d-253d94e9b4d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:22.594041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.593958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ece67d8a-3382-414a-b84d-253d94e9b4d7" (UID: "ece67d8a-3382-414a-b84d-253d94e9b4d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:22.594041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.594021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ece67d8a-3382-414a-b84d-253d94e9b4d7" (UID: "ece67d8a-3382-414a-b84d-253d94e9b4d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.595661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.595639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg" (OuterVolumeSpecName: "kube-api-access-gkcxg") pod "ece67d8a-3382-414a-b84d-253d94e9b4d7" (UID: "ece67d8a-3382-414a-b84d-253d94e9b4d7"). InnerVolumeSpecName "kube-api-access-gkcxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:22.595661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.595644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ece67d8a-3382-414a-b84d-253d94e9b4d7" (UID: "ece67d8a-3382-414a-b84d-253d94e9b4d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:22.607896 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.607869    2578 generic.go:356] "Generic (PLEG): container finished" podID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerID="1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7" exitCode=0
Sep 16 19:16:22.608021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.607934    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"
Sep 16 19:16:22.608021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.607938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerDied","Data":"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"}
Sep 16 19:16:22.608021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.607995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc" event={"ID":"ece67d8a-3382-414a-b84d-253d94e9b4d7","Type":"ContainerDied","Data":"ff726a951d4494b7337d132596f1088ad95d96b21588f5135e7026c6acfbc556"}
Sep 16 19:16:22.608021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.608021    2578 scope.go:117] "RemoveContainer" containerID="cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"
Sep 16 19:16:22.616556 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.616537    2578 scope.go:117] "RemoveContainer" containerID="1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"
Sep 16 19:16:22.624334 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.624313    2578 scope.go:117] "RemoveContainer" containerID="d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"
Sep 16 19:16:22.630739 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.630713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:16:22.631995 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.631971    2578 scope.go:117] "RemoveContainer" containerID="cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"
Sep 16 19:16:22.632303 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:22.632282    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e\": container with ID starting with cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e not found: ID does not exist" containerID="cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"
Sep 16 19:16:22.632363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.632313    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e"} err="failed to get container status \"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e\": rpc error: code = NotFound desc = could not find container \"cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e\": container with ID starting with cd961e52b2070082cead35798d5c15a4937fd69e8fe8350f2c3c31a0cae3e47e not found: ID does not exist"
Sep 16 19:16:22.632363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.632330    2578 scope.go:117] "RemoveContainer" containerID="1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"
Sep 16 19:16:22.632587 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:22.632569    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7\": container with ID starting with 1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7 not found: ID does not exist" containerID="1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"
Sep 16 19:16:22.632652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.632598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7"} err="failed to get container status \"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7\": rpc error: code = NotFound desc = could not find container \"1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7\": container with ID starting with 1ab2d74c25a9178cdbe68af97dce7efc616401430c049ffe04379d024674eda7 not found: ID does not exist"
Sep 16 19:16:22.632652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.632620    2578 scope.go:117] "RemoveContainer" containerID="d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"
Sep 16 19:16:22.632898 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:22.632876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2\": container with ID starting with d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2 not found: ID does not exist" containerID="d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"
Sep 16 19:16:22.632978 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.632903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2"} err="failed to get container status \"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2\": rpc error: code = NotFound desc = could not find container \"d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2\": container with ID starting with d34ccd049990ce747ed98120c6defc47cdd2bfa4cbff2177a4f8329f82fbddf2 not found: ID does not exist"
Sep 16 19:16:22.635726 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.635701    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ccc95dd4-9qxdc"]
Sep 16 19:16:22.694605 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.694569    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.694605 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.694601    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.694767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.694614    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ece67d8a-3382-414a-b84d-253d94e9b4d7-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.694767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.694628    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcxg\" (UniqueName: \"kubernetes.io/projected/ece67d8a-3382-414a-b84d-253d94e9b4d7-kube-api-access-gkcxg\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.694767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:22.694639    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ece67d8a-3382-414a-b84d-253d94e9b4d7-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:16:23.025560 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025480    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:16:23.025778 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025766    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="storage-initializer"
Sep 16 19:16:23.025819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="storage-initializer"
Sep 16 19:16:23.025819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025796    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025806    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container"
Sep 16 19:16:23.025819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025811    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025836    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025841    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025867    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="storage-initializer"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025873    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="storage-initializer"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025932    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kserve-container"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025949    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kube-rbac-proxy"
Sep 16 19:16:23.025989 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.025955    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c38604-f70b-4e26-9446-6d729cdd9297" containerName="kserve-container"
Sep 16 19:16:23.030400 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.030382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.036642 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.036548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:16:23.036642 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.036577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 19:16:23.036784 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.036548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:16:23.036784 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.036753    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:16:23.037049 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.037029    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:23.037214 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.037201    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:23.037402 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.037391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:16:23.045552 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.045529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:16:23.079053 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.079025    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:16:23.082336 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.082321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.085734 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.085719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:16:23.096021 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.096001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:16:23.097665 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.097642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.097756 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.097684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.097756 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.097741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbjc\" (UniqueName: \"kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.097834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.097776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.097834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.097826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.198692 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.198899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.198899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.198899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.199078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.199078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.199078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.198986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69rl\" (UniqueName: \"kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.199078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.199078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.199344 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbjc\" (UniqueName: \"kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.199344 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:23.199178    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 19:16:23.199344 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:23.199244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls podName:4f069b97-4cd2-4a27-bb10-49b9b8075b31 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:23.69922684 +0000 UTC m=+1842.670704070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls") pod "isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 16 19:16:23.199344 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.199545 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.199545 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.199477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.208293 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.208271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbjc\" (UniqueName: \"kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.299984 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.299903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.299984 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.299940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.299984 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.299961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.299984 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.299979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69rl\" (UniqueName: \"kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.300276 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.300028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.300276 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:23.300059    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:16:23.300276 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:16:23.300124    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls podName:8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:23.80010889 +0000 UTC m=+1842.771586117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls") pod "isvc-canary-rollback-predictor-75f85bc9f9-p48md" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:16:23.300438 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.300369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.300643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.300627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.300679 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.300632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.309710 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.309688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69rl\" (UniqueName: \"kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.603206 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.603177    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece67d8a-3382-414a-b84d-253d94e9b4d7" path="/var/lib/kubelet/pods/ece67d8a-3382-414a-b84d-253d94e9b4d7/volumes"
Sep 16 19:16:23.702069 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.702015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.704495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.704476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.802921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.802878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.805427 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.805398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-p48md\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:23.940492 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.940407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:23.992502 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:23.992160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:24.216524 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.216493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:16:24.218641 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:16:24.218610    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f069b97_4cd2_4a27_bb10_49b9b8075b31.slice/crio-44ce3e7782543d825f6141ee5e9d2caa01bef72ae4aa4e18e793921a316b2401 WatchSource:0}: Error finding container 44ce3e7782543d825f6141ee5e9d2caa01bef72ae4aa4e18e793921a316b2401: Status 404 returned error can't find the container with id 44ce3e7782543d825f6141ee5e9d2caa01bef72ae4aa4e18e793921a316b2401
Sep 16 19:16:24.279148 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.279048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:16:24.281216 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:16:24.281189    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7320e5_ad8a_4e90_a18a_0b2d30b9fc55.slice/crio-9b9d0c2f65495b2fa61301a9f0235479fb8a3bdce4a4c4b620f32140bcc3ee7a WatchSource:0}: Error finding container 9b9d0c2f65495b2fa61301a9f0235479fb8a3bdce4a4c4b620f32140bcc3ee7a: Status 404 returned error can't find the container with id 9b9d0c2f65495b2fa61301a9f0235479fb8a3bdce4a4c4b620f32140bcc3ee7a
Sep 16 19:16:24.620070 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.620034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerStarted","Data":"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"}
Sep 16 19:16:24.620555 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.620097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerStarted","Data":"9b9d0c2f65495b2fa61301a9f0235479fb8a3bdce4a4c4b620f32140bcc3ee7a"}
Sep 16 19:16:24.621960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.621931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerStarted","Data":"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"}
Sep 16 19:16:24.622069 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:24.621992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerStarted","Data":"44ce3e7782543d825f6141ee5e9d2caa01bef72ae4aa4e18e793921a316b2401"}
Sep 16 19:16:28.639124 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:28.639093    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerID="c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4" exitCode=0
Sep 16 19:16:28.639555 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:28.639171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerDied","Data":"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"}
Sep 16 19:16:28.640885 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:28.640844    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerID="08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e" exitCode=0
Sep 16 19:16:28.640968 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:28.640918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerDied","Data":"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"}
Sep 16 19:16:29.647518 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.647483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerStarted","Data":"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"}
Sep 16 19:16:29.647976 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.647547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerStarted","Data":"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"}
Sep 16 19:16:29.647976 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.647746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:29.649683 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.649661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerStarted","Data":"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"}
Sep 16 19:16:29.649835 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.649716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerStarted","Data":"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"}
Sep 16 19:16:29.649993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.649976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:29.670408 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.670366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podStartSLOduration=2.981756176 podStartE2EDuration="6.670353768s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688597592s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:29.669456787 +0000 UTC m=+1848.640934045" watchObservedRunningTime="2026-09-16 19:16:29.670353768 +0000 UTC m=+1848.641831015"
Sep 16 19:16:29.695082 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:29.695040    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podStartSLOduration=2.871391364 podStartE2EDuration="6.695027163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823635799s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:29.693511116 +0000 UTC m=+1848.664988381" watchObservedRunningTime="2026-09-16 19:16:29.695027163 +0000 UTC m=+1848.666504410"
Sep 16 19:16:30.654337 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:30.654306    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:30.654717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:30.654362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:30.655299 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:30.655268    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:16:30.655394 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:30.655277    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:16:31.657930 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:31.657884    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:16:31.658402 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:31.658044    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:16:36.663161 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:36.663132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:16:36.663686 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:36.663633    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:16:36.664157 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:36.664142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:16:36.664624 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:36.664602    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:16:46.663660 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:46.663618    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:16:46.664819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:46.664794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:16:48.880184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.880129    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:16:48.883621 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.883597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:48.886921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.886899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 19:16:48.887055 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.886940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:16:48.900177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.900147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:16:48.919636 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.919606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:48.919785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.919656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:48.919785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.919692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:48.919785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.919761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:48.920004 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:48.919786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdr89\" (UniqueName: \"kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021113 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021113 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdr89\" (UniqueName: \"kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021375 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.021672 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.021650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.022028 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.022006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.022076 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.022006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.023743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.023717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.031346 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.031316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdr89\" (UniqueName: \"kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.194507 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.194419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:49.473553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.473530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:16:49.475429 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:16:49.475397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5586667f_5a65_4f84_b786_590c726e575f.slice/crio-76c70de3719a371a0eed32f532f91856b853110c2064a8595fe165024745a0f9 WatchSource:0}: Error finding container 76c70de3719a371a0eed32f532f91856b853110c2064a8595fe165024745a0f9: Status 404 returned error can't find the container with id 76c70de3719a371a0eed32f532f91856b853110c2064a8595fe165024745a0f9
Sep 16 19:16:49.739156 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.739052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerStarted","Data":"49ee4d7b2580059060358b8f67b70ce2a2c504036721c07dea3fb13878f2f76d"}
Sep 16 19:16:49.739156 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:49.739137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerStarted","Data":"76c70de3719a371a0eed32f532f91856b853110c2064a8595fe165024745a0f9"}
Sep 16 19:16:54.762628 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:54.762588    2578 generic.go:356] "Generic (PLEG): container finished" podID="5586667f-5a65-4f84-b786-590c726e575f" containerID="49ee4d7b2580059060358b8f67b70ce2a2c504036721c07dea3fb13878f2f76d" exitCode=0
Sep 16 19:16:54.763059 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:54.762662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerDied","Data":"49ee4d7b2580059060358b8f67b70ce2a2c504036721c07dea3fb13878f2f76d"}
Sep 16 19:16:54.763905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:54.763890    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:16:56.663723 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:56.663679    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:16:56.664939 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:56.664910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:16:58.788397 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.788308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerStarted","Data":"476b739bcca0af2da9cbd2fc4ddc95232eb4e5d4a99d06ec3a357ab0256e5511"}
Sep 16 19:16:58.788397 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.788369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerStarted","Data":"7116250a35cf7b8f28c411cb7452e9a6289ee8213e0e3697325e480df98c4d7e"}
Sep 16 19:16:58.788877 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.788688    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:58.788877 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.788818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:16:58.790088 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.790059    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:16:58.812029 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:58.811972    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podStartSLOduration=2.587538589 podStartE2EDuration="10.811957578s" totalImagesPullingTime="3.616130146s" totalInitContainerRuntime="4.608288843s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:58.810114726 +0000 UTC m=+1877.781591975" watchObservedRunningTime="2026-09-16 19:16:58.811957578 +0000 UTC m=+1877.783434826"
Sep 16 19:16:59.792703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:16:59.792663    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:17:04.797649 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:04.797561    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:17:04.798265 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:04.798237    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:17:06.663929 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:06.663882    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:17:06.665073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:06.665042    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:17:14.798534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:14.798499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:17:16.668153 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:16.668110    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:17:16.668600 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:16.668228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:17:26.664586 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:26.664552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:17:26.665844 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:26.664553    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:17:30.956257 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:30.956218    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:17:30.957119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:30.957052    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container" containerID="cri-o://7116250a35cf7b8f28c411cb7452e9a6289ee8213e0e3697325e480df98c4d7e" gracePeriod=30
Sep 16 19:17:30.957241 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:30.957092    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" containerID="cri-o://476b739bcca0af2da9cbd2fc4ddc95232eb4e5d4a99d06ec3a357ab0256e5511" gracePeriod=30
Sep 16 19:17:31.939979 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:31.939941    2578 generic.go:356] "Generic (PLEG): container finished" podID="5586667f-5a65-4f84-b786-590c726e575f" containerID="476b739bcca0af2da9cbd2fc4ddc95232eb4e5d4a99d06ec3a357ab0256e5511" exitCode=2
Sep 16 19:17:31.940167 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:31.940016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerDied","Data":"476b739bcca0af2da9cbd2fc4ddc95232eb4e5d4a99d06ec3a357ab0256e5511"}
Sep 16 19:17:34.793779 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:34.793737    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:36.664420 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:36.664385    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:17:36.664892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:36.664524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:17:38.598499 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:38.598456    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:17:39.793108 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:39.793067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:44.793029 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:44.792982    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:44.793487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:44.793136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:17:46.665819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:46.665791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:17:46.810361 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:46.810326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:17:46.810630 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:46.810603    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container" containerID="cri-o://75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed" gracePeriod=30
Sep 16 19:17:46.810710 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:46.810658    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kube-rbac-proxy" containerID="cri-o://65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039" gracePeriod=30
Sep 16 19:17:46.811567 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:46.811548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:17:47.009575 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:47.009534    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerID="65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039" exitCode=2
Sep 16 19:17:47.009742 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:47.009603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerDied","Data":"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"}
Sep 16 19:17:49.793932 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:49.793890    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:51.648298 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.648267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:17:51.750895 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.750778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location\") pod \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") "
Sep 16 19:17:51.750895 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.750821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") pod \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") "
Sep 16 19:17:51.750895 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.750877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert\") pod \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") "
Sep 16 19:17:51.751182 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.750909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config\") pod \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") "
Sep 16 19:17:51.751182 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.750945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f69rl\" (UniqueName: \"kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl\") pod \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\" (UID: \"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55\") "
Sep 16 19:17:51.751182 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.751125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.751352 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.751251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.751352 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.751321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.753146 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.753123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:51.753317 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.753293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl" (OuterVolumeSpecName: "kube-api-access-f69rl") pod "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" (UID: "8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55"). InnerVolumeSpecName "kube-api-access-f69rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:51.851611 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.851575    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69rl\" (UniqueName: \"kubernetes.io/projected/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-api-access-f69rl\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.851611 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.851606    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.851611 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.851616    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.851829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.851627    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.851829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:51.851636    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.032554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.032475    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerID="75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed" exitCode=0
Sep 16 19:17:52.032554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.032538    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"
Sep 16 19:17:52.032727 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.032557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerDied","Data":"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"}
Sep 16 19:17:52.032727 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.032621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md" event={"ID":"8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55","Type":"ContainerDied","Data":"9b9d0c2f65495b2fa61301a9f0235479fb8a3bdce4a4c4b620f32140bcc3ee7a"}
Sep 16 19:17:52.032727 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.032649    2578 scope.go:117] "RemoveContainer" containerID="65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"
Sep 16 19:17:52.041438 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.041424    2578 scope.go:117] "RemoveContainer" containerID="75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"
Sep 16 19:17:52.049279 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.049262    2578 scope.go:117] "RemoveContainer" containerID="c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"
Sep 16 19:17:52.057328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.057301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:17:52.057891 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.057875    2578 scope.go:117] "RemoveContainer" containerID="65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"
Sep 16 19:17:52.058230 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:52.058192    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039\": container with ID starting with 65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039 not found: ID does not exist" containerID="65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"
Sep 16 19:17:52.058358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.058230    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039"} err="failed to get container status \"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039\": rpc error: code = NotFound desc = could not find container \"65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039\": container with ID starting with 65aecc5b460d88ea20014c87fce093254f31afaeaadb64073b5db9bbb8d6e039 not found: ID does not exist"
Sep 16 19:17:52.058358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.058253    2578 scope.go:117] "RemoveContainer" containerID="75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"
Sep 16 19:17:52.058683 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:52.058610    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed\": container with ID starting with 75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed not found: ID does not exist" containerID="75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"
Sep 16 19:17:52.058683 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.058643    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed"} err="failed to get container status \"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed\": rpc error: code = NotFound desc = could not find container \"75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed\": container with ID starting with 75e64e21d1c960a79c3427b00032661cfe87cd7e9f12ad423d2cbfa0e64bb3ed not found: ID does not exist"
Sep 16 19:17:52.058683 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.058667    2578 scope.go:117] "RemoveContainer" containerID="c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"
Sep 16 19:17:52.059033 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:52.059011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4\": container with ID starting with c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4 not found: ID does not exist" containerID="c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"
Sep 16 19:17:52.059104 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.059043    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4"} err="failed to get container status \"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4\": rpc error: code = NotFound desc = could not find container \"c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4\": container with ID starting with c6dacb80354e54a8f648d9cdb33ccd2f589c316a09c822570fed74bef4d760a4 not found: ID does not exist"
Sep 16 19:17:52.060428 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:52.060408    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-p48md"]
Sep 16 19:17:53.054223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.054191    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:17:53.054706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.054404    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" containerID="cri-o://f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9" gracePeriod=30
Sep 16 19:17:53.054706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.054492    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kube-rbac-proxy" containerID="cri-o://517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab" gracePeriod=30
Sep 16 19:17:53.147182 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:17:53.147472 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147461    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container"
Sep 16 19:17:53.147520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147475    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container"
Sep 16 19:17:53.147520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kube-rbac-proxy"
Sep 16 19:17:53.147520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147492    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kube-rbac-proxy"
Sep 16 19:17:53.147520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="storage-initializer"
Sep 16 19:17:53.147520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147519    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="storage-initializer"
Sep 16 19:17:53.147687 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kserve-container"
Sep 16 19:17:53.147687 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.147583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" containerName="kube-rbac-proxy"
Sep 16 19:17:53.152005 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.151990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.154954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.154932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:17:53.163198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.163165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:17:53.263878 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.263801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.264054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.263909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnlk\" (UniqueName: \"kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.264054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.263944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.264054 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.263966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.264246 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.264065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365402 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365574 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365574 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnlk\" (UniqueName: \"kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365574 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365574 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.365948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.365928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.366220 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.366197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.366265 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.366223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.367907 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.367887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.375128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.375107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnlk\" (UniqueName: \"kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk\") pod \"isvc-canary-rollback-predictor-75f85bc9f9-pf82r\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.461963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.461932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:53.603085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.603054    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55" path="/var/lib/kubelet/pods/8a7320e5-ad8a-4e90-a18a-0b2d30b9fc55/volumes"
Sep 16 19:17:53.735658 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:53.735631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:17:53.738027 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:17:53.737996    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcdde3c_a019_4a10_8d0b_4cd030611ad9.slice/crio-d0bd951963316e3a4730f20fe51e6e7b5b7ce2b94e160b31c3dc9a5dbbec5ea5 WatchSource:0}: Error finding container d0bd951963316e3a4730f20fe51e6e7b5b7ce2b94e160b31c3dc9a5dbbec5ea5: Status 404 returned error can't find the container with id d0bd951963316e3a4730f20fe51e6e7b5b7ce2b94e160b31c3dc9a5dbbec5ea5
Sep 16 19:17:54.046517 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:54.046422    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerID="517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab" exitCode=2
Sep 16 19:17:54.046517 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:54.046485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerDied","Data":"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"}
Sep 16 19:17:54.048259 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:54.048239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerStarted","Data":"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"}
Sep 16 19:17:54.048393 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:54.048282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerStarted","Data":"d0bd951963316e3a4730f20fe51e6e7b5b7ce2b94e160b31c3dc9a5dbbec5ea5"}
Sep 16 19:17:54.793371 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:54.793327    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:56.658918 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:56.658845    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 16 19:17:56.664992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:56.664969    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:17:58.001630 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.001605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:17:58.068169 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.068130    2578 generic.go:356] "Generic (PLEG): container finished" podID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerID="fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f" exitCode=0
Sep 16 19:17:58.068331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.068208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerDied","Data":"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"}
Sep 16 19:17:58.070255 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.070233    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerID="f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9" exitCode=0
Sep 16 19:17:58.070391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.070275    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"
Sep 16 19:17:58.070391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.070308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerDied","Data":"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"}
Sep 16 19:17:58.070391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.070364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv" event={"ID":"4f069b97-4cd2-4a27-bb10-49b9b8075b31","Type":"ContainerDied","Data":"44ce3e7782543d825f6141ee5e9d2caa01bef72ae4aa4e18e793921a316b2401"}
Sep 16 19:17:58.070391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.070391    2578 scope.go:117] "RemoveContainer" containerID="517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"
Sep 16 19:17:58.081815 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.081797    2578 scope.go:117] "RemoveContainer" containerID="f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"
Sep 16 19:17:58.095503 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.095486    2578 scope.go:117] "RemoveContainer" containerID="08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"
Sep 16 19:17:58.103693 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.103673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location\") pod \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") "
Sep 16 19:17:58.103772 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.103702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config\") pod \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") "
Sep 16 19:17:58.103772 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.103750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shbjc\" (UniqueName: \"kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc\") pod \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") "
Sep 16 19:17:58.103892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.103776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert\") pod \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") "
Sep 16 19:17:58.103954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.103902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") pod \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\" (UID: \"4f069b97-4cd2-4a27-bb10-49b9b8075b31\") "
Sep 16 19:17:58.104082 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104058    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f069b97-4cd2-4a27-bb10-49b9b8075b31" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:58.104190 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f069b97-4cd2-4a27-bb10-49b9b8075b31" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:58.104256 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f069b97-4cd2-4a27-bb10-49b9b8075b31" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:58.104338 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104322    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:58.104397 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104344    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:58.104397 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.104362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:58.105831 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.105812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc" (OuterVolumeSpecName: "kube-api-access-shbjc") pod "4f069b97-4cd2-4a27-bb10-49b9b8075b31" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31"). InnerVolumeSpecName "kube-api-access-shbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:58.105965 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.105886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f069b97-4cd2-4a27-bb10-49b9b8075b31" (UID: "4f069b97-4cd2-4a27-bb10-49b9b8075b31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:58.112456 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.112434    2578 scope.go:117] "RemoveContainer" containerID="517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"
Sep 16 19:17:58.112787 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:58.112766    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab\": container with ID starting with 517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab not found: ID does not exist" containerID="517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"
Sep 16 19:17:58.112899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.112794    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab"} err="failed to get container status \"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab\": rpc error: code = NotFound desc = could not find container \"517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab\": container with ID starting with 517072fa03a8ac1027f5719b6720c9c35798d24c08f9e0645cbeaaf6d22d60ab not found: ID does not exist"
Sep 16 19:17:58.112899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.112812    2578 scope.go:117] "RemoveContainer" containerID="f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"
Sep 16 19:17:58.113124 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:58.113106    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9\": container with ID starting with f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9 not found: ID does not exist" containerID="f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"
Sep 16 19:17:58.113205 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.113129    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9"} err="failed to get container status \"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9\": rpc error: code = NotFound desc = could not find container \"f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9\": container with ID starting with f6cd2db7a23a6b25690098e233efd0ed0f5beed55e389165169d43a1371bd7d9 not found: ID does not exist"
Sep 16 19:17:58.113205 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.113144    2578 scope.go:117] "RemoveContainer" containerID="08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"
Sep 16 19:17:58.113399 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:17:58.113380    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e\": container with ID starting with 08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e not found: ID does not exist" containerID="08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"
Sep 16 19:17:58.113464 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.113408    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e"} err="failed to get container status \"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e\": rpc error: code = NotFound desc = could not find container \"08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e\": container with ID starting with 08b4c919dd08fe4ba81c8d238160a279142a9c729bf16fb849508542115eed6e not found: ID does not exist"
Sep 16 19:17:58.204759 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.204730    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f069b97-4cd2-4a27-bb10-49b9b8075b31-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:58.204759 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.204761    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbjc\" (UniqueName: \"kubernetes.io/projected/4f069b97-4cd2-4a27-bb10-49b9b8075b31-kube-api-access-shbjc\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:17:58.394753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.394719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:17:58.402314 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:58.402288    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-c557dd7db-nnbpv"]
Sep 16 19:17:59.078111 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.078063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerStarted","Data":"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"}
Sep 16 19:17:59.078576 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.078145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerStarted","Data":"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"}
Sep 16 19:17:59.078576 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.078345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:17:59.105349 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.105292    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podStartSLOduration=2.3682388100000002 podStartE2EDuration="6.10527405s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.73703524s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:59.103211667 +0000 UTC m=+1938.074688941" watchObservedRunningTime="2026-09-16 19:17:59.10527405 +0000 UTC m=+1938.076751298"
Sep 16 19:17:59.603237 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.603202    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" path="/var/lib/kubelet/pods/4f069b97-4cd2-4a27-bb10-49b9b8075b31/volumes"
Sep 16 19:17:59.793275 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:17:59.793228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:18:00.084260 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:00.084167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:18:00.085565 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:00.085534    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:01.090949 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.090903    2578 generic.go:356] "Generic (PLEG): container finished" podID="5586667f-5a65-4f84-b786-590c726e575f" containerID="7116250a35cf7b8f28c411cb7452e9a6289ee8213e0e3697325e480df98c4d7e" exitCode=137
Sep 16 19:18:01.091348 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.090983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerDied","Data":"7116250a35cf7b8f28c411cb7452e9a6289ee8213e0e3697325e480df98c4d7e"}
Sep 16 19:18:01.091544 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.091521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:01.601778 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.601751    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:18:01.632525 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.632496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls\") pod \"5586667f-5a65-4f84-b786-590c726e575f\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") "
Sep 16 19:18:01.632706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.632543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config\") pod \"5586667f-5a65-4f84-b786-590c726e575f\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") "
Sep 16 19:18:01.632706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.632586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location\") pod \"5586667f-5a65-4f84-b786-590c726e575f\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") "
Sep 16 19:18:01.632706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.632626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert\") pod \"5586667f-5a65-4f84-b786-590c726e575f\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") "
Sep 16 19:18:01.632706 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.632651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdr89\" (UniqueName: \"kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89\") pod \"5586667f-5a65-4f84-b786-590c726e575f\" (UID: \"5586667f-5a65-4f84-b786-590c726e575f\") "
Sep 16 19:18:01.633361 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.633319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5586667f-5a65-4f84-b786-590c726e575f" (UID: "5586667f-5a65-4f84-b786-590c726e575f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:01.633361 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.633343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5586667f-5a65-4f84-b786-590c726e575f" (UID: "5586667f-5a65-4f84-b786-590c726e575f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:01.634943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.634915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5586667f-5a65-4f84-b786-590c726e575f" (UID: "5586667f-5a65-4f84-b786-590c726e575f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:01.635404 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.635380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89" (OuterVolumeSpecName: "kube-api-access-gdr89") pod "5586667f-5a65-4f84-b786-590c726e575f" (UID: "5586667f-5a65-4f84-b786-590c726e575f"). InnerVolumeSpecName "kube-api-access-gdr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:01.644184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.644144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5586667f-5a65-4f84-b786-590c726e575f" (UID: "5586667f-5a65-4f84-b786-590c726e575f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:01.733329 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.733241    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5586667f-5a65-4f84-b786-590c726e575f-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:18:01.733329 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.733274    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:18:01.733329 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.733284    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5586667f-5a65-4f84-b786-590c726e575f-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:18:01.733329 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.733294    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5586667f-5a65-4f84-b786-590c726e575f-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:18:01.733329 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:01.733303    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdr89\" (UniqueName: \"kubernetes.io/projected/5586667f-5a65-4f84-b786-590c726e575f-kube-api-access-gdr89\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:18:02.097601 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.097570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"
Sep 16 19:18:02.098083 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.097570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4" event={"ID":"5586667f-5a65-4f84-b786-590c726e575f","Type":"ContainerDied","Data":"76c70de3719a371a0eed32f532f91856b853110c2064a8595fe165024745a0f9"}
Sep 16 19:18:02.098083 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.097706    2578 scope.go:117] "RemoveContainer" containerID="476b739bcca0af2da9cbd2fc4ddc95232eb4e5d4a99d06ec3a357ab0256e5511"
Sep 16 19:18:02.106386 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.106366    2578 scope.go:117] "RemoveContainer" containerID="7116250a35cf7b8f28c411cb7452e9a6289ee8213e0e3697325e480df98c4d7e"
Sep 16 19:18:02.114179 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.114161    2578 scope.go:117] "RemoveContainer" containerID="49ee4d7b2580059060358b8f67b70ce2a2c504036721c07dea3fb13878f2f76d"
Sep 16 19:18:02.120641 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.120616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:18:02.125335 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:02.125309    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5qcc4"]
Sep 16 19:18:03.603144 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:03.603113    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5586667f-5a65-4f84-b786-590c726e575f" path="/var/lib/kubelet/pods/5586667f-5a65-4f84-b786-590c726e575f/volumes"
Sep 16 19:18:06.095723 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:06.095691    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:18:06.096145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:06.096055    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:16.096135 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:16.096090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:26.096746 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:26.096704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:36.096786 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:36.096692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:46.097041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:46.097002    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:18:56.096993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:18:56.096952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:19:06.097077 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:06.097031    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:19:16.097014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:16.096987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:19:23.318282 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:23.318245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:19:23.318704 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:23.318478    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" containerID="cri-o://25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80" gracePeriod=30
Sep 16 19:19:23.318704 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:23.318509    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kube-rbac-proxy" containerID="cri-o://155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c" gracePeriod=30
Sep 16 19:19:23.472434 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:23.472403    2578 generic.go:356] "Generic (PLEG): container finished" podID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerID="155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c" exitCode=2
Sep 16 19:19:23.472584 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:23.472471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerDied","Data":"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"}
Sep 16 19:19:26.092130 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:26.092085    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 19:19:26.097043 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:26.097012    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:19:28.066390 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.066365    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:19:28.159703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.159669    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config\") pod \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") "
Sep 16 19:19:28.159894 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.159755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls\") pod \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") "
Sep 16 19:19:28.159894 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.159779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location\") pod \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") "
Sep 16 19:19:28.159894 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.159800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert\") pod \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") "
Sep 16 19:19:28.159894 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.159832    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpnlk\" (UniqueName: \"kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk\") pod \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\" (UID: \"0dcdde3c-a019-4a10-8d0b-4cd030611ad9\") "
Sep 16 19:19:28.160145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.160107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0dcdde3c-a019-4a10-8d0b-4cd030611ad9" (UID: "0dcdde3c-a019-4a10-8d0b-4cd030611ad9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:28.160145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.160114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0dcdde3c-a019-4a10-8d0b-4cd030611ad9" (UID: "0dcdde3c-a019-4a10-8d0b-4cd030611ad9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:28.160228 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.160147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0dcdde3c-a019-4a10-8d0b-4cd030611ad9" (UID: "0dcdde3c-a019-4a10-8d0b-4cd030611ad9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:28.162048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.162022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk" (OuterVolumeSpecName: "kube-api-access-cpnlk") pod "0dcdde3c-a019-4a10-8d0b-4cd030611ad9" (UID: "0dcdde3c-a019-4a10-8d0b-4cd030611ad9"). InnerVolumeSpecName "kube-api-access-cpnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:28.162121 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.162052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0dcdde3c-a019-4a10-8d0b-4cd030611ad9" (UID: "0dcdde3c-a019-4a10-8d0b-4cd030611ad9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:28.260631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.260600    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnlk\" (UniqueName: \"kubernetes.io/projected/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-api-access-cpnlk\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.260631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.260626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.260631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.260636    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.260864 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.260645    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.260864 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.260653    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdde3c-a019-4a10-8d0b-4cd030611ad9-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.442155 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:19:28.442401 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="storage-initializer"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="storage-initializer"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442422    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442454    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="storage-initializer"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="storage-initializer"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442465    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="storage-initializer"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442471    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="storage-initializer"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442482    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442497    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442505    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442515    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442520    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442582    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5586667f-5a65-4f84-b786-590c726e575f" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442588    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442595    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kserve-container"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442601    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f069b97-4cd2-4a27-bb10-49b9b8075b31" containerName="kube-rbac-proxy"
Sep 16 19:19:28.442652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.442608    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerName="kube-rbac-proxy"
Sep 16 19:19:28.445901 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.445884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.449537 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.449516    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:19:28.450233 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.450218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 19:19:28.462984 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.462958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:19:28.477287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.477266    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:19:28.480487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.480470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.491083 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.491064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:19:28.495190 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.495165    2578 generic.go:356] "Generic (PLEG): container finished" podID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" containerID="25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80" exitCode=0
Sep 16 19:19:28.495278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.495224    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"
Sep 16 19:19:28.495278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.495224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerDied","Data":"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"}
Sep 16 19:19:28.495367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.495287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r" event={"ID":"0dcdde3c-a019-4a10-8d0b-4cd030611ad9","Type":"ContainerDied","Data":"d0bd951963316e3a4730f20fe51e6e7b5b7ce2b94e160b31c3dc9a5dbbec5ea5"}
Sep 16 19:19:28.495367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.495327    2578 scope.go:117] "RemoveContainer" containerID="155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"
Sep 16 19:19:28.503534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.503515    2578 scope.go:117] "RemoveContainer" containerID="25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"
Sep 16 19:19:28.511781 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.511762    2578 scope.go:117] "RemoveContainer" containerID="fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"
Sep 16 19:19:28.512613 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.512596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:19:28.521325 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.521308    2578 scope.go:117] "RemoveContainer" containerID="155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"
Sep 16 19:19:28.521649 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:19:28.521622    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c\": container with ID starting with 155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c not found: ID does not exist" containerID="155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"
Sep 16 19:19:28.521740 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.521661    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c"} err="failed to get container status \"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c\": rpc error: code = NotFound desc = could not find container \"155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c\": container with ID starting with 155159a3406e931da47c3ae26cd1faa56a5f06c6d3f24461d067068d7737cb2c not found: ID does not exist"
Sep 16 19:19:28.521740 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.521685    2578 scope.go:117] "RemoveContainer" containerID="25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"
Sep 16 19:19:28.522009 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:19:28.521992    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80\": container with ID starting with 25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80 not found: ID does not exist" containerID="25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"
Sep 16 19:19:28.522059 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.522016    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80"} err="failed to get container status \"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80\": rpc error: code = NotFound desc = could not find container \"25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80\": container with ID starting with 25ed12084563f3a2b4d7657ebf6ac03d05d913de40786645d99452d5347f9e80 not found: ID does not exist"
Sep 16 19:19:28.522059 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.522032    2578 scope.go:117] "RemoveContainer" containerID="fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"
Sep 16 19:19:28.522293 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:19:28.522272    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f\": container with ID starting with fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f not found: ID does not exist" containerID="fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"
Sep 16 19:19:28.522376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.522298    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f"} err="failed to get container status \"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f\": rpc error: code = NotFound desc = could not find container \"fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f\": container with ID starting with fa11fdc2c76d33c6834259a11f7a9e9aac1ff9d1709dd64626b64a5b929f596f not found: ID does not exist"
Sep 16 19:19:28.527222 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.527201    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:19:28.529468 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.529447    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-75f85bc9f9-pf82r"]
Sep 16 19:19:28.562869 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.562823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnjk\" (UniqueName: \"kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.562993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.562879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.562993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.562927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.562993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.562953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.562993 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.562978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.563197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.563018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwcl\" (UniqueName: \"kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.563197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.563044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.563197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.563066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.563197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.563086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.563197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.563122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.663686 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.663686 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.663911 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnjk\" (UniqueName: \"kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.663911 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.663911 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.663911 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:19:28.663923    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.663975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwcl\" (UniqueName: \"kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:19:28.663989    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls podName:30195e99-c495-47ab-88e4-b8162d89345b nodeName:}" failed. No retries permitted until 2026-09-16 19:19:29.163968555 +0000 UTC m=+2028.135445788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls") pod "isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" (UID: "30195e99-c495-47ab-88e4-b8162d89345b") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.664128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.664431 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.664556 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.664667 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.664667 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.664897 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.664846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.665047 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.665032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.666377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.666355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.674181 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.674163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnjk\" (UniqueName: \"kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk\") pod \"isvc-canary-stop-predictor-6cbd7d9f59-4ktdz\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:28.674409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.674393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwcl\" (UniqueName: \"kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:28.790180 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:28.790104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:29.063027 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.063003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:19:29.065094 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:19:29.065059    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2891a2c_d0fc_4497_95ca_e09b8c84cea6.slice/crio-9c7cf0630c3c1198914ce0eaf500e72b7b2e68f571aca1a427138ec79414d212 WatchSource:0}: Error finding container 9c7cf0630c3c1198914ce0eaf500e72b7b2e68f571aca1a427138ec79414d212: Status 404 returned error can't find the container with id 9c7cf0630c3c1198914ce0eaf500e72b7b2e68f571aca1a427138ec79414d212
Sep 16 19:19:29.168223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.168186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:29.170645 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.170623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:29.355774 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.355739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:29.503438 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.503398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerStarted","Data":"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"}
Sep 16 19:19:29.503669 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.503461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerStarted","Data":"9c7cf0630c3c1198914ce0eaf500e72b7b2e68f571aca1a427138ec79414d212"}
Sep 16 19:19:29.602730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.602696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcdde3c-a019-4a10-8d0b-4cd030611ad9" path="/var/lib/kubelet/pods/0dcdde3c-a019-4a10-8d0b-4cd030611ad9/volumes"
Sep 16 19:19:29.665748 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:29.665703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:19:29.666197 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:19:29.666168    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30195e99_c495_47ab_88e4_b8162d89345b.slice/crio-7051a99beca5f46aabdf8c23a947f1323a18ba021b7717899e9adb3b13f67d63 WatchSource:0}: Error finding container 7051a99beca5f46aabdf8c23a947f1323a18ba021b7717899e9adb3b13f67d63: Status 404 returned error can't find the container with id 7051a99beca5f46aabdf8c23a947f1323a18ba021b7717899e9adb3b13f67d63
Sep 16 19:19:30.509223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:30.509187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerStarted","Data":"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"}
Sep 16 19:19:30.509605 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:30.509243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerStarted","Data":"7051a99beca5f46aabdf8c23a947f1323a18ba021b7717899e9adb3b13f67d63"}
Sep 16 19:19:33.527478 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:33.527442    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerID="d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967" exitCode=0
Sep 16 19:19:33.527873 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:33.527519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerDied","Data":"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"}
Sep 16 19:19:34.535179 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.535141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerStarted","Data":"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"}
Sep 16 19:19:34.535651 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.535214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerStarted","Data":"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"}
Sep 16 19:19:34.535651 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.535438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:34.536817 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.536791    2578 generic.go:356] "Generic (PLEG): container finished" podID="30195e99-c495-47ab-88e4-b8162d89345b" containerID="b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69" exitCode=0
Sep 16 19:19:34.536982 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.536832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerDied","Data":"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"}
Sep 16 19:19:34.561576 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:34.561533    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podStartSLOduration=2.748998283 podStartE2EDuration="6.561517944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812519661s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:34.559974041 +0000 UTC m=+2033.531451289" watchObservedRunningTime="2026-09-16 19:19:34.561517944 +0000 UTC m=+2033.532995194"
Sep 16 19:19:35.543419 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.543378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerStarted","Data":"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"}
Sep 16 19:19:35.543905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.543630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:35.543905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.543677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerStarted","Data":"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"}
Sep 16 19:19:35.543905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.543895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:35.544150 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.543944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:35.544997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.544970    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:19:35.545122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.545058    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:19:35.567980 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:35.567928    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podStartSLOduration=3.670471848 podStartE2EDuration="7.567914113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.897442265s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:35.566169805 +0000 UTC m=+2034.537647058" watchObservedRunningTime="2026-09-16 19:19:35.567914113 +0000 UTC m=+2034.539391361"
Sep 16 19:19:36.548787 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:36.548734    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:19:36.549253 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:36.548793    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:19:41.554442 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:41.554407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:19:41.555013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:41.554519    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:19:41.555100 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:41.555021    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:19:41.555250 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:41.555219    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:19:51.555724 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:51.555675    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:19:51.556228 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:19:51.555675    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:01.555970 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:01.555925    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:01.556392 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:01.555925    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:20:11.554967 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:11.554879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:11.555392 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:11.554888    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:20:21.555408 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:21.555363    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:21.555886 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:21.555363    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:20:31.555514 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:31.555477    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:20:31.555951 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:31.555475    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:41.555731 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:41.555694    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:20:41.556116 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:41.555742    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:20:49.381486 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:49.381452    2578 scope.go:117] "RemoveContainer" containerID="376fab93912e3255fbcfa4989209fc85bd1b4bcd4be88599e30862c5661f9569"
Sep 16 19:20:49.389724 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:49.389703    2578 scope.go:117] "RemoveContainer" containerID="796913601f0bb76f4a348b6c5b80dbce9185eae89dcc6e8cce51da30c85abf28"
Sep 16 19:20:49.397384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:49.397362    2578 scope.go:117] "RemoveContainer" containerID="70dd906d3b279f87fc106441fb0ca415dc14503abd1c442a39696c948ed2e144"
Sep 16 19:20:51.556002 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:51.555973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:20:51.556491 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:51.556062    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:20:58.439470 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.439414    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:20:58.440032 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.439703    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" containerID="cri-o://3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913" gracePeriod=30
Sep 16 19:20:58.440032 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.439811    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kube-rbac-proxy" containerID="cri-o://bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395" gracePeriod=30
Sep 16 19:20:58.512573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.512542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:20:58.512795 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.512758    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" containerID="cri-o://cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1" gracePeriod=30
Sep 16 19:20:58.512911 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.512800    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kube-rbac-proxy" containerID="cri-o://d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8" gracePeriod=30
Sep 16 19:20:58.892088 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.892057    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerID="d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8" exitCode=2
Sep 16 19:20:58.892277 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.892125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerDied","Data":"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"}
Sep 16 19:20:58.894323 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.894303    2578 generic.go:356] "Generic (PLEG): container finished" podID="30195e99-c495-47ab-88e4-b8162d89345b" containerID="bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395" exitCode=2
Sep 16 19:20:58.894421 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:20:58.894360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerDied","Data":"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"}
Sep 16 19:21:01.549619 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:01.549575    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 16 19:21:01.550046 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:01.549587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:21:01.555066 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:01.555045    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:21:01.555129 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:01.555048    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:21:03.536744 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.536720    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:21:03.623998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.623971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config\") pod \"30195e99-c495-47ab-88e4-b8162d89345b\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") "
Sep 16 19:21:03.624197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") pod \"30195e99-c495-47ab-88e4-b8162d89345b\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") "
Sep 16 19:21:03.624197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert\") pod \"30195e99-c495-47ab-88e4-b8162d89345b\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") "
Sep 16 19:21:03.624197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624091    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location\") pod \"30195e99-c495-47ab-88e4-b8162d89345b\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") "
Sep 16 19:21:03.624197 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtwcl\" (UniqueName: \"kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl\") pod \"30195e99-c495-47ab-88e4-b8162d89345b\" (UID: \"30195e99-c495-47ab-88e4-b8162d89345b\") "
Sep 16 19:21:03.624404 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30195e99-c495-47ab-88e4-b8162d89345b" (UID: "30195e99-c495-47ab-88e4-b8162d89345b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:03.624472 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30195e99-c495-47ab-88e4-b8162d89345b" (UID: "30195e99-c495-47ab-88e4-b8162d89345b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:03.624472 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30195e99-c495-47ab-88e4-b8162d89345b" (UID: "30195e99-c495-47ab-88e4-b8162d89345b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:03.624599 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624534    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.624599 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624550    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30195e99-c495-47ab-88e4-b8162d89345b-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.624599 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.624569    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30195e99-c495-47ab-88e4-b8162d89345b-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.626529 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.626505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30195e99-c495-47ab-88e4-b8162d89345b" (UID: "30195e99-c495-47ab-88e4-b8162d89345b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:03.626529 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.626517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl" (OuterVolumeSpecName: "kube-api-access-vtwcl") pod "30195e99-c495-47ab-88e4-b8162d89345b" (UID: "30195e99-c495-47ab-88e4-b8162d89345b"). InnerVolumeSpecName "kube-api-access-vtwcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:03.644591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.644568    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:21:03.724929 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.724887    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location\") pod \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") "
Sep 16 19:21:03.725122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.724948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config\") pod \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") "
Sep 16 19:21:03.725122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.724971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls\") pod \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") "
Sep 16 19:21:03.725122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsnjk\" (UniqueName: \"kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk\") pod \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") "
Sep 16 19:21:03.725122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert\") pod \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\" (UID: \"d2891a2c-d0fc-4497-95ca-e09b8c84cea6\") "
Sep 16 19:21:03.725342 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2891a2c-d0fc-4497-95ca-e09b8c84cea6" (UID: "d2891a2c-d0fc-4497-95ca-e09b8c84cea6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:03.725342 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2891a2c-d0fc-4497-95ca-e09b8c84cea6" (UID: "d2891a2c-d0fc-4497-95ca-e09b8c84cea6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:03.725453 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725385    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwcl\" (UniqueName: \"kubernetes.io/projected/30195e99-c495-47ab-88e4-b8162d89345b-kube-api-access-vtwcl\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.725453 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725411    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.725453 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725427    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30195e99-c495-47ab-88e4-b8162d89345b-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.725596 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.725521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2891a2c-d0fc-4497-95ca-e09b8c84cea6" (UID: "d2891a2c-d0fc-4497-95ca-e09b8c84cea6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:03.727218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.727200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk" (OuterVolumeSpecName: "kube-api-access-xsnjk") pod "d2891a2c-d0fc-4497-95ca-e09b8c84cea6" (UID: "d2891a2c-d0fc-4497-95ca-e09b8c84cea6"). InnerVolumeSpecName "kube-api-access-xsnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:03.727304 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.727285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2891a2c-d0fc-4497-95ca-e09b8c84cea6" (UID: "d2891a2c-d0fc-4497-95ca-e09b8c84cea6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:03.826147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.826110    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.826147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.826140    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.826147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.826151    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnjk\" (UniqueName: \"kubernetes.io/projected/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-kube-api-access-xsnjk\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.826378 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.826160    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2891a2c-d0fc-4497-95ca-e09b8c84cea6-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:21:03.915966 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.915841    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerID="cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1" exitCode=0
Sep 16 19:21:03.915966 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.915937    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"
Sep 16 19:21:03.916201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.915931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerDied","Data":"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"}
Sep 16 19:21:03.916201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.916102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz" event={"ID":"d2891a2c-d0fc-4497-95ca-e09b8c84cea6","Type":"ContainerDied","Data":"9c7cf0630c3c1198914ce0eaf500e72b7b2e68f571aca1a427138ec79414d212"}
Sep 16 19:21:03.916201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.916146    2578 scope.go:117] "RemoveContainer" containerID="d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"
Sep 16 19:21:03.917960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.917936    2578 generic.go:356] "Generic (PLEG): container finished" podID="30195e99-c495-47ab-88e4-b8162d89345b" containerID="3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913" exitCode=0
Sep 16 19:21:03.918091 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.917960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerDied","Data":"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"}
Sep 16 19:21:03.918091 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.917996    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"
Sep 16 19:21:03.918091 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.918027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76" event={"ID":"30195e99-c495-47ab-88e4-b8162d89345b","Type":"ContainerDied","Data":"7051a99beca5f46aabdf8c23a947f1323a18ba021b7717899e9adb3b13f67d63"}
Sep 16 19:21:03.926712 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.926692    2578 scope.go:117] "RemoveContainer" containerID="cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"
Sep 16 19:21:03.935271 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.935245    2578 scope.go:117] "RemoveContainer" containerID="d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"
Sep 16 19:21:03.943273 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.943250    2578 scope.go:117] "RemoveContainer" containerID="d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"
Sep 16 19:21:03.943581 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.943560    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8\": container with ID starting with d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8 not found: ID does not exist" containerID="d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"
Sep 16 19:21:03.943636 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.943592    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8"} err="failed to get container status \"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8\": rpc error: code = NotFound desc = could not find container \"d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8\": container with ID starting with d2a11487393cb9de06bee410b8c71615af0161d9ae405164d0725f0a85254bb8 not found: ID does not exist"
Sep 16 19:21:03.943636 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.943613    2578 scope.go:117] "RemoveContainer" containerID="cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"
Sep 16 19:21:03.943961 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.943842    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1\": container with ID starting with cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1 not found: ID does not exist" containerID="cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"
Sep 16 19:21:03.943961 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.943943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1"} err="failed to get container status \"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1\": rpc error: code = NotFound desc = could not find container \"cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1\": container with ID starting with cfa7d30b01b8efb192fc3c347933f01052a2dfb9f8495094c1e004cceb119ea1 not found: ID does not exist"
Sep 16 19:21:03.944101 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.943967    2578 scope.go:117] "RemoveContainer" containerID="d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"
Sep 16 19:21:03.944223 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.944207    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967\": container with ID starting with d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967 not found: ID does not exist" containerID="d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"
Sep 16 19:21:03.944309 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.944229    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967"} err="failed to get container status \"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967\": rpc error: code = NotFound desc = could not find container \"d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967\": container with ID starting with d32307a2bcbcf755deb369a8dd6382cd53838b6b4935be93d0e7b9879d6fb967 not found: ID does not exist"
Sep 16 19:21:03.944309 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.944243    2578 scope.go:117] "RemoveContainer" containerID="bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"
Sep 16 19:21:03.947637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.947616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:21:03.950909 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.950883    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6cbd7d9f59-4ktdz"]
Sep 16 19:21:03.952842 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.952823    2578 scope.go:117] "RemoveContainer" containerID="3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"
Sep 16 19:21:03.962814 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.962790    2578 scope.go:117] "RemoveContainer" containerID="b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"
Sep 16 19:21:03.968726 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.968700    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:21:03.971492 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.971474    2578 scope.go:117] "RemoveContainer" containerID="bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"
Sep 16 19:21:03.971824 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.971801    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395\": container with ID starting with bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395 not found: ID does not exist" containerID="bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"
Sep 16 19:21:03.971892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.971836    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395"} err="failed to get container status \"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395\": rpc error: code = NotFound desc = could not find container \"bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395\": container with ID starting with bb160c8e8d5aaba2fe3a26d74def23e3bea057b32408a7d465c666ca66219395 not found: ID does not exist"
Sep 16 19:21:03.971892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.971880    2578 scope.go:117] "RemoveContainer" containerID="3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"
Sep 16 19:21:03.972248 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.972163    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913\": container with ID starting with 3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913 not found: ID does not exist" containerID="3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"
Sep 16 19:21:03.972248 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.972219    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913"} err="failed to get container status \"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913\": rpc error: code = NotFound desc = could not find container \"3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913\": container with ID starting with 3a9c70e70f64f7344a650a46989530f49b380e0289dd995f969827eab7b2d913 not found: ID does not exist"
Sep 16 19:21:03.972248 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.972235    2578 scope.go:117] "RemoveContainer" containerID="b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"
Sep 16 19:21:03.972559 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:21:03.972543    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69\": container with ID starting with b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69 not found: ID does not exist" containerID="b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"
Sep 16 19:21:03.972731 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.972563    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69"} err="failed to get container status \"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69\": rpc error: code = NotFound desc = could not find container \"b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69\": container with ID starting with b50613c8305a374e7a67cc5b9bb717bc18b32170a318ecc7d5aba7fbc818fd69 not found: ID does not exist"
Sep 16 19:21:03.976398 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:03.976379    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-64b9b9d9c4-phr76"]
Sep 16 19:21:05.602708 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:05.602671    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30195e99-c495-47ab-88e4-b8162d89345b" path="/var/lib/kubelet/pods/30195e99-c495-47ab-88e4-b8162d89345b/volumes"
Sep 16 19:21:05.603187 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:05.603171    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" path="/var/lib/kubelet/pods/d2891a2c-d0fc-4497-95ca-e09b8c84cea6/volumes"
Sep 16 19:21:08.780386 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780355    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:21:08.780844 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780820    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container"
Sep 16 19:21:08.780844 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780838    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="storage-initializer"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="storage-initializer"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780901    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="storage-initializer"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780910    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="storage-initializer"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780921    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780930    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780948    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kube-rbac-proxy"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kube-rbac-proxy"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kube-rbac-proxy"
Sep 16 19:21:08.780997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.780985    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kube-rbac-proxy"
Sep 16 19:21:08.781480 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.781071    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kube-rbac-proxy"
Sep 16 19:21:08.781480 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.781084    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kserve-container"
Sep 16 19:21:08.781480 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.781095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2891a2c-d0fc-4497-95ca-e09b8c84cea6" containerName="kserve-container"
Sep 16 19:21:08.781480 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.781104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30195e99-c495-47ab-88e4-b8162d89345b" containerName="kube-rbac-proxy"
Sep 16 19:21:08.785963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.785941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.791324 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.791287    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 19:21:08.791839 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.791814    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:21:08.792042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.791818    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:21:08.793407 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.793299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:21:08.793530 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.793504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:21:08.793598 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.793559    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 16 19:21:08.793663 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.793635    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:21:08.802407 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.802381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:21:08.859591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.859558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.859744 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.859602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.859744 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.859625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.859744 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.859646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.859744 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.859697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2md\" (UniqueName: \"kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960189 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2md\" (UniqueName: \"kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.960769 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.961014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.960994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.961077 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.961028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.962558 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.962540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:08.973302 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:08.973278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2md\" (UniqueName: \"kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md\") pod \"isvc-lightgbm-predictor-54f85ddf88-wxbq5\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:09.098529 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:09.098495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:09.382066 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:09.381949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:21:09.384670 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:21:09.384629    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be5b7c4_9387_4443_849a_08d5e80240f3.slice/crio-bb0d8c6476af6777c96de1ee2878cd7be19b435fee1b55336016fe821c755c90 WatchSource:0}: Error finding container bb0d8c6476af6777c96de1ee2878cd7be19b435fee1b55336016fe821c755c90: Status 404 returned error can't find the container with id bb0d8c6476af6777c96de1ee2878cd7be19b435fee1b55336016fe821c755c90
Sep 16 19:21:09.946104 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:09.946065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerStarted","Data":"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"}
Sep 16 19:21:09.946436 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:09.946135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerStarted","Data":"bb0d8c6476af6777c96de1ee2878cd7be19b435fee1b55336016fe821c755c90"}
Sep 16 19:21:13.965997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:13.965963    2578 generic.go:356] "Generic (PLEG): container finished" podID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerID="aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2" exitCode=0
Sep 16 19:21:13.966362 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:13.966047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerDied","Data":"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"}
Sep 16 19:21:21.002712 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:21.002678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerStarted","Data":"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"}
Sep 16 19:21:21.003103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:21.002739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerStarted","Data":"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"}
Sep 16 19:21:21.003103 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:21.002920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:21.031597 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:21.031530    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podStartSLOduration=2.93975675 podStartE2EDuration="13.031509483s" totalImagesPullingTime="6.116616475s" totalInitContainerRuntime="3.975136258s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:21.029662999 +0000 UTC m=+2140.001140248" watchObservedRunningTime="2026-09-16 19:21:21.031509483 +0000 UTC m=+2140.002986732"
Sep 16 19:21:22.007406 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:22.007372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:22.008684 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:22.008655    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:21:23.011378 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:23.011338    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:21:28.016180 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:28.016141    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:21:28.016752 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:28.016730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:21:38.016910 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:38.016817    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:21:48.017081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:48.017036    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:21:58.016823 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:21:58.016783    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:08.016936 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:08.016895    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:18.017586 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:18.017540    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:28.017679 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:28.017636    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:33.598649 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:33.598598    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:43.603223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:43.603193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:22:48.824136 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:48.824096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:22:48.824607 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:48.824498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" containerID="cri-o://15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162" gracePeriod=30
Sep 16 19:22:48.824791 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:48.824737    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kube-rbac-proxy" containerID="cri-o://a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac" gracePeriod=30
Sep 16 19:22:49.366029 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:49.365990    2578 generic.go:356] "Generic (PLEG): container finished" podID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerID="a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac" exitCode=2
Sep 16 19:22:49.366274 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:49.366050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerDied","Data":"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"}
Sep 16 19:22:53.011545 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:53.011506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:22:53.598950 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:53.598904    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:22:53.967314 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:53.967288    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:22:54.074832 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.074800    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075108    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075119    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075135    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kube-rbac-proxy"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075140    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kube-rbac-proxy"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075148    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="storage-initializer"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="storage-initializer"
Sep 16 19:22:54.075198 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075197    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kserve-container"
Sep 16 19:22:54.075440 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.075207    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerName="kube-rbac-proxy"
Sep 16 19:22:54.077317 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.077301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.080367 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.080347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 16 19:22:54.080709 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.080690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:22:54.094089 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.094066    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:22:54.116223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls\") pod \"0be5b7c4-9387-4443-849a-08d5e80240f3\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") "
Sep 16 19:22:54.116223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw2md\" (UniqueName: \"kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md\") pod \"0be5b7c4-9387-4443-849a-08d5e80240f3\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") "
Sep 16 19:22:54.116223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location\") pod \"0be5b7c4-9387-4443-849a-08d5e80240f3\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") "
Sep 16 19:22:54.116741 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert\") pod \"0be5b7c4-9387-4443-849a-08d5e80240f3\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") "
Sep 16 19:22:54.116741 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config\") pod \"0be5b7c4-9387-4443-849a-08d5e80240f3\" (UID: \"0be5b7c4-9387-4443-849a-08d5e80240f3\") "
Sep 16 19:22:54.116872 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116797    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0be5b7c4-9387-4443-849a-08d5e80240f3" (UID: "0be5b7c4-9387-4443-849a-08d5e80240f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:54.116872 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0be5b7c4-9387-4443-849a-08d5e80240f3" (UID: "0be5b7c4-9387-4443-849a-08d5e80240f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.117023 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.116924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0be5b7c4-9387-4443-849a-08d5e80240f3" (UID: "0be5b7c4-9387-4443-849a-08d5e80240f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.118538 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.118519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md" (OuterVolumeSpecName: "kube-api-access-mw2md") pod "0be5b7c4-9387-4443-849a-08d5e80240f3" (UID: "0be5b7c4-9387-4443-849a-08d5e80240f3"). InnerVolumeSpecName "kube-api-access-mw2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:54.118632 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.118612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0be5b7c4-9387-4443-849a-08d5e80240f3" (UID: "0be5b7c4-9387-4443-849a-08d5e80240f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:54.217795 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.217746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.217795 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.217807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2gw\" (UniqueName: \"kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.217894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.217918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.217952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.218014    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.218025    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0be5b7c4-9387-4443-849a-08d5e80240f3-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.218034    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw2md\" (UniqueName: \"kubernetes.io/projected/0be5b7c4-9387-4443-849a-08d5e80240f3-kube-api-access-mw2md\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.218051 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.218048    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be5b7c4-9387-4443-849a-08d5e80240f3-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.218283 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.218061    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0be5b7c4-9387-4443-849a-08d5e80240f3-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.318824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.318788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.318824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.318828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2gw\" (UniqueName: \"kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.318879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.318901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.318941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319094 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:22:54.318989    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:22:54.319094 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:22:54.319078    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls podName:163d14dd-fd0d-4b77-af77-90d920df4d23 nodeName:}" failed. No retries permitted until 2026-09-16 19:22:54.819055852 +0000 UTC m=+2233.790533104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:22:54.319344 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.319318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319595 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.319577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.319641 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.319623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.330166 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.330139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2gw\" (UniqueName: \"kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.389391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.389306    2578 generic.go:356] "Generic (PLEG): container finished" podID="0be5b7c4-9387-4443-849a-08d5e80240f3" containerID="15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162" exitCode=0
Sep 16 19:22:54.389391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.389370    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"
Sep 16 19:22:54.389561 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.389389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerDied","Data":"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"}
Sep 16 19:22:54.389561 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.389455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5" event={"ID":"0be5b7c4-9387-4443-849a-08d5e80240f3","Type":"ContainerDied","Data":"bb0d8c6476af6777c96de1ee2878cd7be19b435fee1b55336016fe821c755c90"}
Sep 16 19:22:54.389561 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.389482    2578 scope.go:117] "RemoveContainer" containerID="a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"
Sep 16 19:22:54.403509 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.403484    2578 scope.go:117] "RemoveContainer" containerID="15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"
Sep 16 19:22:54.411161 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.411130    2578 scope.go:117] "RemoveContainer" containerID="aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"
Sep 16 19:22:54.416370 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.416335    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:22:54.419128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.419104    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-wxbq5"]
Sep 16 19:22:54.419697 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.419682    2578 scope.go:117] "RemoveContainer" containerID="a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"
Sep 16 19:22:54.420010 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:22:54.419988    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac\": container with ID starting with a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac not found: ID does not exist" containerID="a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"
Sep 16 19:22:54.420099 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.420021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac"} err="failed to get container status \"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac\": rpc error: code = NotFound desc = could not find container \"a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac\": container with ID starting with a06c2f604e841db56d6a9e2cd872384a12221246a655fb90179f3908e40d0dac not found: ID does not exist"
Sep 16 19:22:54.420099 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.420047    2578 scope.go:117] "RemoveContainer" containerID="15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"
Sep 16 19:22:54.420343 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:22:54.420325    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162\": container with ID starting with 15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162 not found: ID does not exist" containerID="15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"
Sep 16 19:22:54.420389 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.420352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162"} err="failed to get container status \"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162\": rpc error: code = NotFound desc = could not find container \"15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162\": container with ID starting with 15fd9d25cddd4e83cb245900a8d4698471f3b76c3627efc563f9ac00cb0fd162 not found: ID does not exist"
Sep 16 19:22:54.420389 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.420369    2578 scope.go:117] "RemoveContainer" containerID="aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"
Sep 16 19:22:54.420820 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:22:54.420802    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2\": container with ID starting with aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2 not found: ID does not exist" containerID="aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"
Sep 16 19:22:54.420919 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.420826    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2"} err="failed to get container status \"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2\": rpc error: code = NotFound desc = could not find container \"aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2\": container with ID starting with aea2c7b3c15b8fad152c1f8a639f9ea060a54f9b1c95e63f36a037419fba5dd2 not found: ID does not exist"
Sep 16 19:22:54.821497 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.821389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.823924 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.823896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:54.986401 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:54.986355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:22:55.261026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:55.261002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:22:55.266249 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:22:55.266217    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163d14dd_fd0d_4b77_af77_90d920df4d23.slice/crio-a3a28bf8d324d781f399b51bd022df46b0cbbef027284a869537f0476a635c41 WatchSource:0}: Error finding container a3a28bf8d324d781f399b51bd022df46b0cbbef027284a869537f0476a635c41: Status 404 returned error can't find the container with id a3a28bf8d324d781f399b51bd022df46b0cbbef027284a869537f0476a635c41
Sep 16 19:22:55.396894 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:55.396823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerStarted","Data":"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"}
Sep 16 19:22:55.397061 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:55.396921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerStarted","Data":"a3a28bf8d324d781f399b51bd022df46b0cbbef027284a869537f0476a635c41"}
Sep 16 19:22:55.603081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:55.603054    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be5b7c4-9387-4443-849a-08d5e80240f3" path="/var/lib/kubelet/pods/0be5b7c4-9387-4443-849a-08d5e80240f3/volumes"
Sep 16 19:22:59.414707 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:59.414670    2578 generic.go:356] "Generic (PLEG): container finished" podID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerID="f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332" exitCode=0
Sep 16 19:22:59.415161 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:22:59.414750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerDied","Data":"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"}
Sep 16 19:23:00.421671 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.421635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerStarted","Data":"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"}
Sep 16 19:23:00.422092 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.421729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerStarted","Data":"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"}
Sep 16 19:23:00.422092 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.422035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:23:00.422183 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.422093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:23:00.423383 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.423358    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:00.444907 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:00.444843    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podStartSLOduration=2.780359691 podStartE2EDuration="6.444829916s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.664470225s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:00.443272616 +0000 UTC m=+2239.414749865" watchObservedRunningTime="2026-09-16 19:23:00.444829916 +0000 UTC m=+2239.416307165"
Sep 16 19:23:01.425789 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:01.425748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:06.430704 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:06.430622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:23:06.431192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:06.431157    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:16.431096 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:16.431048    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:26.431788 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:26.431728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:36.432005 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:36.431965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:46.431513 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:46.431471    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:23:56.431866 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:23:56.431819    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:24:06.431635 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:06.431594    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:24:16.432013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:16.431979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:24:24.347115 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:24.347083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:24:24.347566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:24.347320    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" containerID="cri-o://7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687" gracePeriod=30
Sep 16 19:24:24.347566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:24.347367    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kube-rbac-proxy" containerID="cri-o://88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44" gracePeriod=30
Sep 16 19:24:24.773491 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:24.773395    2578 generic.go:356] "Generic (PLEG): container finished" podID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerID="88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44" exitCode=2
Sep 16 19:24:24.773491 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:24.773470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerDied","Data":"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"}
Sep 16 19:24:26.426520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:26.426481    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:24:26.431826 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:26.431799    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:24:29.684531 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.684506    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:24:29.797881 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.797771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert\") pod \"163d14dd-fd0d-4b77-af77-90d920df4d23\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") "
Sep 16 19:24:29.797881 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.797808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk2gw\" (UniqueName: \"kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw\") pod \"163d14dd-fd0d-4b77-af77-90d920df4d23\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") "
Sep 16 19:24:29.797881 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.797842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config\") pod \"163d14dd-fd0d-4b77-af77-90d920df4d23\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") "
Sep 16 19:24:29.798192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.797914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location\") pod \"163d14dd-fd0d-4b77-af77-90d920df4d23\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") "
Sep 16 19:24:29.798192 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.797962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") pod \"163d14dd-fd0d-4b77-af77-90d920df4d23\" (UID: \"163d14dd-fd0d-4b77-af77-90d920df4d23\") "
Sep 16 19:24:29.798305 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.798209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "163d14dd-fd0d-4b77-af77-90d920df4d23" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:29.798439 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.798402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "163d14dd-fd0d-4b77-af77-90d920df4d23" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:29.798585 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.798488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "163d14dd-fd0d-4b77-af77-90d920df4d23" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:29.799930 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.799899    2578 generic.go:356] "Generic (PLEG): container finished" podID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerID="7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687" exitCode=0
Sep 16 19:24:29.800060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.799974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"
Sep 16 19:24:29.800060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.799982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerDied","Data":"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"}
Sep 16 19:24:29.800176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.800060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m" event={"ID":"163d14dd-fd0d-4b77-af77-90d920df4d23","Type":"ContainerDied","Data":"a3a28bf8d324d781f399b51bd022df46b0cbbef027284a869537f0476a635c41"}
Sep 16 19:24:29.800176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.800098    2578 scope.go:117] "RemoveContainer" containerID="88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"
Sep 16 19:24:29.800328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.800301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw" (OuterVolumeSpecName: "kube-api-access-zk2gw") pod "163d14dd-fd0d-4b77-af77-90d920df4d23" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23"). InnerVolumeSpecName "kube-api-access-zk2gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:29.800386 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.800320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "163d14dd-fd0d-4b77-af77-90d920df4d23" (UID: "163d14dd-fd0d-4b77-af77-90d920df4d23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:29.813889 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.813873    2578 scope.go:117] "RemoveContainer" containerID="7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"
Sep 16 19:24:29.820877 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.820842    2578 scope.go:117] "RemoveContainer" containerID="f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"
Sep 16 19:24:29.828039 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828024    2578 scope.go:117] "RemoveContainer" containerID="88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"
Sep 16 19:24:29.828329 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:24:29.828307    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44\": container with ID starting with 88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44 not found: ID does not exist" containerID="88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"
Sep 16 19:24:29.828377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828338    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44"} err="failed to get container status \"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44\": rpc error: code = NotFound desc = could not find container \"88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44\": container with ID starting with 88ab91c380d28edb9297ecaacb52b7631f5b2bf08fce319c31021f0d4ed70a44 not found: ID does not exist"
Sep 16 19:24:29.828377 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828355    2578 scope.go:117] "RemoveContainer" containerID="7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"
Sep 16 19:24:29.828609 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:24:29.828590    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687\": container with ID starting with 7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687 not found: ID does not exist" containerID="7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"
Sep 16 19:24:29.828661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687"} err="failed to get container status \"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687\": rpc error: code = NotFound desc = could not find container \"7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687\": container with ID starting with 7c8c2b71e0ca8ad9970abad6c6576a3a4dabc62b7997525e38273486800fb687 not found: ID does not exist"
Sep 16 19:24:29.828661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828631    2578 scope.go:117] "RemoveContainer" containerID="f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"
Sep 16 19:24:29.828843 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:24:29.828827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332\": container with ID starting with f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332 not found: ID does not exist" containerID="f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"
Sep 16 19:24:29.828906 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.828861    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332"} err="failed to get container status \"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332\": rpc error: code = NotFound desc = could not find container \"f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332\": container with ID starting with f3b36e67c0a811a434b29da13adca152322face4f63c91d58b13ca97456f4332 not found: ID does not exist"
Sep 16 19:24:29.899117 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.899076    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2gw\" (UniqueName: \"kubernetes.io/projected/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-api-access-zk2gw\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:24:29.899117 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.899112    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:24:29.899283 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.899135    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163d14dd-fd0d-4b77-af77-90d920df4d23-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:24:29.899283 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.899145    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163d14dd-fd0d-4b77-af77-90d920df4d23-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:24:29.899283 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:29.899154    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163d14dd-fd0d-4b77-af77-90d920df4d23-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.133347 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:30.133315    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:24:30.137780 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:30.137756    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-rzc5m"]
Sep 16 19:24:31.607958 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:24:31.607926    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" path="/var/lib/kubelet/pods/163d14dd-fd0d-4b77-af77-90d920df4d23/volumes"
Sep 16 19:25:24.901799 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.901763    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902094    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kube-rbac-proxy"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902107    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kube-rbac-proxy"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902132    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="storage-initializer"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="storage-initializer"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902188    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kserve-container"
Sep 16 19:25:24.902287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.902196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="163d14dd-fd0d-4b77-af77-90d920df4d23" containerName="kube-rbac-proxy"
Sep 16 19:25:24.905479 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.905463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:24.909075 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909055    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:24.909195 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 19:25:24.909195 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909056    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:25:24.909926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909906    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:24.909926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909919    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:25:24.910094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:25:24.910094 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.909909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:24.922908 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.922872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:24.954992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.954957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75gtj\" (UniqueName: \"kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:24.955175 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.955006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:24.955175 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.955047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:24.955175 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.955073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:24.955323 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:24.955190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.056598 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.056562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.056598 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.056603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75gtj\" (UniqueName: \"kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.056824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.056636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.056824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.056663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.056824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.056689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.057203 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.057172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.057457 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.057436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.057504 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.057446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.059255 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.059232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.067060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.067035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75gtj\" (UniqueName: \"kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.216413 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.216324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:25.493998 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:25.493971    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:25.496444 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:25:25.496411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce52bc0c_8010_4e35_b4cd_586b3bc6d64e.slice/crio-2e31eb676076344047f555bf9eea1ca4c01276c16bfcba371684dae5f8823c2e WatchSource:0}: Error finding container 2e31eb676076344047f555bf9eea1ca4c01276c16bfcba371684dae5f8823c2e: Status 404 returned error can't find the container with id 2e31eb676076344047f555bf9eea1ca4c01276c16bfcba371684dae5f8823c2e
Sep 16 19:25:26.024648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:26.024611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerStarted","Data":"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"}
Sep 16 19:25:26.025048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:26.024677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerStarted","Data":"2e31eb676076344047f555bf9eea1ca4c01276c16bfcba371684dae5f8823c2e"}
Sep 16 19:25:30.042466 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:30.042423    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerID="e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9" exitCode=0
Sep 16 19:25:30.042882 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:30.042498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerDied","Data":"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"}
Sep 16 19:25:31.048674 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:31.048638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerStarted","Data":"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"}
Sep 16 19:25:31.049125 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:31.048710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerStarted","Data":"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"}
Sep 16 19:25:31.049125 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:31.048978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:31.077080 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:31.077026    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" podStartSLOduration=3.135257507 podStartE2EDuration="7.07700675s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941749243s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:31.074495395 +0000 UTC m=+2390.045972681" watchObservedRunningTime="2026-09-16 19:25:31.07700675 +0000 UTC m=+2390.048483999"
Sep 16 19:25:32.053501 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:32.053458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:32.054703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:32.054674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:25:33.059033 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:33.058993    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:25:38.065137 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:38.065099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:38.065987 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:38.065963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:44.853633 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:44.853588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:44.854069 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:44.853873    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container" containerID="cri-o://51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab" gracePeriod=30
Sep 16 19:25:44.854069 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:44.853905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kube-rbac-proxy" containerID="cri-o://550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9" gracePeriod=30
Sep 16 19:25:45.107947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.107843    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerID="550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9" exitCode=2
Sep 16 19:25:45.107947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.107887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerDied","Data":"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"}
Sep 16 19:25:45.502963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.502939    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:45.632177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location\") pod \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") "
Sep 16 19:25:45.632177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75gtj\" (UniqueName: \"kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj\") pod \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") "
Sep 16 19:25:45.632177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls\") pod \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") "
Sep 16 19:25:45.632177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert\") pod \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") "
Sep 16 19:25:45.632477 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config\") pod \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\" (UID: \"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e\") "
Sep 16 19:25:45.632534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" (UID: "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:45.632587 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632538    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" (UID: "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:45.632647 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.632594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" (UID: "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:45.634411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.634385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj" (OuterVolumeSpecName: "kube-api-access-75gtj") pod "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" (UID: "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e"). InnerVolumeSpecName "kube-api-access-75gtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:45.634524 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.634479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" (UID: "ce52bc0c-8010-4e35-b4cd-586b3bc6d64e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:45.733646 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.733605    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:25:45.733646 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.733639    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75gtj\" (UniqueName: \"kubernetes.io/projected/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-api-access-75gtj\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:25:45.733646 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.733651    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:25:45.733921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.733663    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:25:45.733921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:45.733674    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:25:46.115176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.115140    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerID="51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab" exitCode=0
Sep 16 19:25:46.115665 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.115192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerDied","Data":"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"}
Sep 16 19:25:46.115665 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.115208    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"
Sep 16 19:25:46.115665 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.115243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6" event={"ID":"ce52bc0c-8010-4e35-b4cd-586b3bc6d64e","Type":"ContainerDied","Data":"2e31eb676076344047f555bf9eea1ca4c01276c16bfcba371684dae5f8823c2e"}
Sep 16 19:25:46.115665 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.115268    2578 scope.go:117] "RemoveContainer" containerID="550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"
Sep 16 19:25:46.124534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.124507    2578 scope.go:117] "RemoveContainer" containerID="51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"
Sep 16 19:25:46.132128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.132103    2578 scope.go:117] "RemoveContainer" containerID="e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"
Sep 16 19:25:46.139640 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.139621    2578 scope.go:117] "RemoveContainer" containerID="550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"
Sep 16 19:25:46.140073 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:25:46.140011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9\": container with ID starting with 550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9 not found: ID does not exist" containerID="550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"
Sep 16 19:25:46.140073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.140046    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9"} err="failed to get container status \"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9\": rpc error: code = NotFound desc = could not find container \"550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9\": container with ID starting with 550b98f8fa0cdd20a42272c31aa2420ea253934486c3f1eac11a72a1c921a8b9 not found: ID does not exist"
Sep 16 19:25:46.140073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.140068    2578 scope.go:117] "RemoveContainer" containerID="51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"
Sep 16 19:25:46.140479 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:25:46.140446    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab\": container with ID starting with 51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab not found: ID does not exist" containerID="51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"
Sep 16 19:25:46.140591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.140483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab"} err="failed to get container status \"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab\": rpc error: code = NotFound desc = could not find container \"51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab\": container with ID starting with 51f54ba29496a8774bb564c76b636be73a2ddef0edc1297f1bc61c853d69d3ab not found: ID does not exist"
Sep 16 19:25:46.140591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.140507    2578 scope.go:117] "RemoveContainer" containerID="e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"
Sep 16 19:25:46.141117 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:25:46.140894    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9\": container with ID starting with e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9 not found: ID does not exist" containerID="e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"
Sep 16 19:25:46.141117 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.140933    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9"} err="failed to get container status \"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9\": rpc error: code = NotFound desc = could not find container \"e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9\": container with ID starting with e3aee0b3087bf14db9088878c09fa81cfac1076bd94372a34d6787b0908d69e9 not found: ID does not exist"
Sep 16 19:25:46.142878 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.142843    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:46.144893 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:46.144870    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-d2rp6"]
Sep 16 19:25:47.603460 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:25:47.603428    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" path="/var/lib/kubelet/pods/ce52bc0c-8010-4e35-b4cd-586b3bc6d64e/volumes"
Sep 16 19:26:00.237393 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237359    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237671    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="storage-initializer"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237682    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="storage-initializer"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kube-rbac-proxy"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kube-rbac-proxy"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237714    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237719    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237766    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kube-rbac-proxy"
Sep 16 19:26:00.237905 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.237776    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce52bc0c-8010-4e35-b4cd-586b3bc6d64e" containerName="kserve-container"
Sep 16 19:26:00.242336 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.242320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.247300 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.247270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:26:00.247300 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.247296    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:26:00.247504 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.247299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 19:26:00.248921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.248668    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:26:00.248921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.248729    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:26:00.248921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.248762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:26:00.249060 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.248978    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:26:00.253670 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.253639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwjh\" (UniqueName: \"kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.253799 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.253709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.253799 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.253771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.253949 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.253804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.253949 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.253835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.255439 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.255416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:26:00.354218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwjh\" (UniqueName: \"kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354411 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:26:00.354346    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:26:00.354694 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:26:00.354429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls podName:3565721d-5e0d-4e31-86c7-3d3c3a1676d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:26:00.854409407 +0000 UTC m=+2419.825886634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-2wkmp" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:26:00.354694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354942 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.354988 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.354924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.364055 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.364026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwjh\" (UniqueName: \"kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.856401 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.856367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:00.858892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:00.858869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-2wkmp\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:01.152020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:01.151933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:01.426932 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:01.426906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:26:01.429421 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:26:01.429386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3565721d_5e0d_4e31_86c7_3d3c3a1676d0.slice/crio-f86f602438cc2806d9ad6ea0145c45f1daddbb4b18ff5ee2930ffd6d1300a1ef WatchSource:0}: Error finding container f86f602438cc2806d9ad6ea0145c45f1daddbb4b18ff5ee2930ffd6d1300a1ef: Status 404 returned error can't find the container with id f86f602438cc2806d9ad6ea0145c45f1daddbb4b18ff5ee2930ffd6d1300a1ef
Sep 16 19:26:02.183868 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:02.183812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerStarted","Data":"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"}
Sep 16 19:26:02.184036 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:02.183898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerStarted","Data":"f86f602438cc2806d9ad6ea0145c45f1daddbb4b18ff5ee2930ffd6d1300a1ef"}
Sep 16 19:26:06.200420 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:06.200383    2578 generic.go:356] "Generic (PLEG): container finished" podID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerID="9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f" exitCode=0
Sep 16 19:26:06.200796 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:06.200451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerDied","Data":"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"}
Sep 16 19:26:06.201583 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:06.201563    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:23.285899 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:23.285843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerStarted","Data":"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"}
Sep 16 19:26:23.286270 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:23.285936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerStarted","Data":"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"}
Sep 16 19:26:23.286270 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:23.286118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:23.310871 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:23.310798    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podStartSLOduration=2.823676526 podStartE2EDuration="23.310781446s" totalImagesPullingTime="16.426683576s" totalInitContainerRuntime="4.060421344s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:23.309151667 +0000 UTC m=+2442.280628928" watchObservedRunningTime="2026-09-16 19:26:23.310781446 +0000 UTC m=+2442.282258696"
Sep 16 19:26:24.289937 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:24.289898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:24.291159 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:24.291129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:26:25.294227 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:25.294181    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:26:30.299277 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:30.299243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:26:30.299834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:30.299805    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:26:40.300105 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:40.300059    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:26:50.300119 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:26:50.300075    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:27:00.300280 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:00.300244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:27:10.300581 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:10.300541    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:27:20.300566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:20.300524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:27:30.301039 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:30.301005    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:27:40.346445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:40.346368    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:27:40.347200 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:40.346647    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container" containerID="cri-o://e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27" gracePeriod=30
Sep 16 19:27:40.347200 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:40.346710    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kube-rbac-proxy" containerID="cri-o://52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4" gracePeriod=30
Sep 16 19:27:40.599703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:40.599621    2578 generic.go:356] "Generic (PLEG): container finished" podID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerID="52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4" exitCode=2
Sep 16 19:27:40.599703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:40.599685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerDied","Data":"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"}
Sep 16 19:27:43.883090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:43.883059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:27:44.018765 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.018662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") pod \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") "
Sep 16 19:27:44.019112 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.019090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjwjh\" (UniqueName: \"kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh\") pod \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") "
Sep 16 19:27:44.019247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.019233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert\") pod \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") "
Sep 16 19:27:44.019391 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.019379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location\") pod \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") "
Sep 16 19:27:44.019490 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.019478    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config\") pod \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\" (UID: \"3565721d-5e0d-4e31-86c7-3d3c3a1676d0\") "
Sep 16 19:27:44.020249 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.020223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3565721d-5e0d-4e31-86c7-3d3c3a1676d0" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:44.024658 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.024384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3565721d-5e0d-4e31-86c7-3d3c3a1676d0" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:44.027043 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.026833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh" (OuterVolumeSpecName: "kube-api-access-mjwjh") pod "3565721d-5e0d-4e31-86c7-3d3c3a1676d0" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0"). InnerVolumeSpecName "kube-api-access-mjwjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:44.027454 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.027256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3565721d-5e0d-4e31-86c7-3d3c3a1676d0" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:44.027661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.027624    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3565721d-5e0d-4e31-86c7-3d3c3a1676d0" (UID: "3565721d-5e0d-4e31-86c7-3d3c3a1676d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:44.120604 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.120567    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:27:44.120604 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.120596    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:27:44.120813 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.120614    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:27:44.120813 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.120626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwjh\" (UniqueName: \"kubernetes.io/projected/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-kube-api-access-mjwjh\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:27:44.120813 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.120635    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3565721d-5e0d-4e31-86c7-3d3c3a1676d0-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:27:44.618006 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.617967    2578 generic.go:356] "Generic (PLEG): container finished" podID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerID="e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27" exitCode=0
Sep 16 19:27:44.618207 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.618058    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"
Sep 16 19:27:44.618207 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.618056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerDied","Data":"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"}
Sep 16 19:27:44.618207 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.618133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp" event={"ID":"3565721d-5e0d-4e31-86c7-3d3c3a1676d0","Type":"ContainerDied","Data":"f86f602438cc2806d9ad6ea0145c45f1daddbb4b18ff5ee2930ffd6d1300a1ef"}
Sep 16 19:27:44.618207 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.618172    2578 scope.go:117] "RemoveContainer" containerID="52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"
Sep 16 19:27:44.626730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.626707    2578 scope.go:117] "RemoveContainer" containerID="e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"
Sep 16 19:27:44.635553 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.635528    2578 scope.go:117] "RemoveContainer" containerID="9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"
Sep 16 19:27:44.643308 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643290    2578 scope.go:117] "RemoveContainer" containerID="52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"
Sep 16 19:27:44.643379 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643320    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:27:44.643614 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:27:44.643595    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4\": container with ID starting with 52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4 not found: ID does not exist" containerID="52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"
Sep 16 19:27:44.643676 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643622    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4"} err="failed to get container status \"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4\": rpc error: code = NotFound desc = could not find container \"52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4\": container with ID starting with 52190fdacf6c607ab2aeb66f710b4222d1c4e30b6491f32fa5307d5006cfdea4 not found: ID does not exist"
Sep 16 19:27:44.643676 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643640    2578 scope.go:117] "RemoveContainer" containerID="e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"
Sep 16 19:27:44.643971 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:27:44.643948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27\": container with ID starting with e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27 not found: ID does not exist" containerID="e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"
Sep 16 19:27:44.644042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643977    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27"} err="failed to get container status \"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27\": rpc error: code = NotFound desc = could not find container \"e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27\": container with ID starting with e56dafc8dedeefcc2a07d40206b41f6730a34fea7b19b7df29416b2123f47f27 not found: ID does not exist"
Sep 16 19:27:44.644042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.643998    2578 scope.go:117] "RemoveContainer" containerID="9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"
Sep 16 19:27:44.644275 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:27:44.644258    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f\": container with ID starting with 9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f not found: ID does not exist" containerID="9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"
Sep 16 19:27:44.644353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.644282    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f"} err="failed to get container status \"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f\": rpc error: code = NotFound desc = could not find container \"9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f\": container with ID starting with 9ff450c9ded2ed6e067c6d316da89bf265c433871145141c534278e85fd5a86f not found: ID does not exist"
Sep 16 19:27:44.646890 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:44.646865    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-2wkmp"]
Sep 16 19:27:45.527603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527570    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527903    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="storage-initializer"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527917    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="storage-initializer"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527929    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kube-rbac-proxy"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527935    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kube-rbac-proxy"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527946    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.527951    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.528020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kserve-container"
Sep 16 19:27:45.528067 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.528027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" containerName="kube-rbac-proxy"
Sep 16 19:27:45.531317 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.531290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.534120 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.534096    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 16 19:27:45.534249 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.534097    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:27:45.535341 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.535314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:27:45.535455 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.535333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:27:45.535455 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.535357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 16 19:27:45.535455 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.535344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:27:45.543525 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.543502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:27:45.603062 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.603032    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3565721d-5e0d-4e31-86c7-3d3c3a1676d0" path="/var/lib/kubelet/pods/3565721d-5e0d-4e31-86c7-3d3c3a1676d0/volumes"
Sep 16 19:27:45.633191 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.633160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.633354 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.633219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmqw\" (UniqueName: \"kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.633354 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.633277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.633354 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.633307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.734187 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.734156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmqw\" (UniqueName: \"kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.734353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.734212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.734353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.734258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.734353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.734321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.734663 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.734633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.735074 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.735058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.736969 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.736947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.743441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.743420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmqw\" (UniqueName: \"kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw\") pod \"isvc-sklearn-s3-predictor-776585ccf5-f7bgq\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:45.841715 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:45.841685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:46.115454 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:46.115428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:27:46.117571 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:27:46.117535    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3331de_3f17_41f9_9ee7_4216a4e22da1.slice/crio-943742ff4b55d3eec57e0e8dd0601ef1b053c7d408c23d5e5d7d08b3aba121db WatchSource:0}: Error finding container 943742ff4b55d3eec57e0e8dd0601ef1b053c7d408c23d5e5d7d08b3aba121db: Status 404 returned error can't find the container with id 943742ff4b55d3eec57e0e8dd0601ef1b053c7d408c23d5e5d7d08b3aba121db
Sep 16 19:27:46.629065 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:46.629020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerStarted","Data":"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"}
Sep 16 19:27:46.629444 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:46.629108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerStarted","Data":"943742ff4b55d3eec57e0e8dd0601ef1b053c7d408c23d5e5d7d08b3aba121db"}
Sep 16 19:27:47.634079 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:47.634040    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerID="367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc" exitCode=0
Sep 16 19:27:47.634079 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:47.634083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerDied","Data":"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"}
Sep 16 19:27:48.641948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:48.641908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerStarted","Data":"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"}
Sep 16 19:27:48.642359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:48.641969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerStarted","Data":"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"}
Sep 16 19:27:48.642359 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:48.642042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:48.665098 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:48.665048    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podStartSLOduration=2.591009371 podStartE2EDuration="3.665032157s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.074022786s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:48.662926653 +0000 UTC m=+2527.634403901" watchObservedRunningTime="2026-09-16 19:27:48.665032157 +0000 UTC m=+2527.636509406"
Sep 16 19:27:49.646522 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:49.646490    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:49.647960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:49.647839    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:27:50.650313 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:50.650272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:27:55.655026 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:55.654996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:27:55.655475 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:27:55.655449    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:05.656147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:05.656100    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:10.562373 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.562339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:28:10.565740 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.565717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.569498 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.569475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 16 19:28:10.569603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.569510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:10.569603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.569510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 16 19:28:10.570325 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.570308    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:10.579708 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.579688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:28:10.646700 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.646669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwqx\" (UniqueName: \"kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.646700 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.646702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.646926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.646724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.646926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.646832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.646926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.646908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.747761 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.747729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.747933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.747771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwqx\" (UniqueName: \"kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.747933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.747788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.747933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.747824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.747933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.747903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.748215 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.748194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.748534 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.748509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.748615 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.748513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.750489 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.750471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.756388 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.756366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwqx\" (UniqueName: \"kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx\") pod \"isvc-paddle-predictor-6d5c9fd4f6-wgw6b\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:10.875765 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:10.875729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:11.147548 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:11.147525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:28:11.149789 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:28:11.149748    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735cbee7_79d4_428a_baa7_dcafc9f951db.slice/crio-33213b6cbff6b3f848f6d56c75ad1f8ee87e52aebbd04c02207e361f39b38454 WatchSource:0}: Error finding container 33213b6cbff6b3f848f6d56c75ad1f8ee87e52aebbd04c02207e361f39b38454: Status 404 returned error can't find the container with id 33213b6cbff6b3f848f6d56c75ad1f8ee87e52aebbd04c02207e361f39b38454
Sep 16 19:28:11.734756 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:11.734717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerStarted","Data":"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"}
Sep 16 19:28:11.735180 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:11.734780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerStarted","Data":"33213b6cbff6b3f848f6d56c75ad1f8ee87e52aebbd04c02207e361f39b38454"}
Sep 16 19:28:15.655438 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:15.655394    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:15.754353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:15.754323    2578 generic.go:356] "Generic (PLEG): container finished" podID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerID="0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50" exitCode=0
Sep 16 19:28:15.754478 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:15.754397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerDied","Data":"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"}
Sep 16 19:28:25.655943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:25.655895    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:26.809645 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.809610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerStarted","Data":"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"}
Sep 16 19:28:26.810090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.809675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerStarted","Data":"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"}
Sep 16 19:28:26.810090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.809965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:26.810179 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.810100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:26.811399 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.811373    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:28:26.836321 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:26.836260    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podStartSLOduration=1.542754325 podStartE2EDuration="16.836239381s" totalImagesPullingTime="10.82987493s" totalInitContainerRuntime="4.463610126s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:26.834961474 +0000 UTC m=+2565.806438727" watchObservedRunningTime="2026-09-16 19:28:26.836239381 +0000 UTC m=+2565.807716631"
Sep 16 19:28:27.814499 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:27.814461    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:28:32.819092 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:32.819062    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:28:32.819593 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:32.819568    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:28:35.655955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:35.655918    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:42.820483 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:42.820441    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:28:45.655662 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:45.655614    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:28:52.820176 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:52.820133    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:28:55.656387 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:28:55.656347    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:29:02.820425 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:02.820334    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:29:04.598557 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:04.598521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:29:12.820039 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:12.820007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:29:14.599609 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:14.599578    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:29:19.658066 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:19.658029    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:29:19.658947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:19.658913    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container" containerID="cri-o://b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425" gracePeriod=30
Sep 16 19:29:19.659767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:19.659315    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kube-rbac-proxy" containerID="cri-o://a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146" gracePeriod=30
Sep 16 19:29:20.034834 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:20.034742    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerID="a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146" exitCode=2
Sep 16 19:29:20.035009 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:20.034824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerDied","Data":"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"}
Sep 16 19:29:20.651486 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:20.651450    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 19:29:21.755005 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:21.754966    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:29:21.755474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:21.755224    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" containerID="cri-o://a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2" gracePeriod=30
Sep 16 19:29:21.755474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:21.755254    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kube-rbac-proxy" containerID="cri-o://d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138" gracePeriod=30
Sep 16 19:29:22.047903 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:22.047786    2578 generic.go:356] "Generic (PLEG): container finished" podID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerID="d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138" exitCode=2
Sep 16 19:29:22.047903 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:22.047836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerDied","Data":"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"}
Sep 16 19:29:22.815652 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:22.815609    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:29:22.819975 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:22.819946    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:29:24.517030 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.517002    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:29:24.529126 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.529103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:29:24.553327 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553296    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xwqx\" (UniqueName: \"kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx\") pod \"735cbee7-79d4-428a-baa7-dcafc9f951db\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") "
Sep 16 19:29:24.553490 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553339    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls\") pod \"735cbee7-79d4-428a-baa7-dcafc9f951db\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") "
Sep 16 19:29:24.553490 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location\") pod \"735cbee7-79d4-428a-baa7-dcafc9f951db\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") "
Sep 16 19:29:24.553490 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553440    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config\") pod \"735cbee7-79d4-428a-baa7-dcafc9f951db\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") "
Sep 16 19:29:24.553490 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert\") pod \"735cbee7-79d4-428a-baa7-dcafc9f951db\" (UID: \"735cbee7-79d4-428a-baa7-dcafc9f951db\") "
Sep 16 19:29:24.553800 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "735cbee7-79d4-428a-baa7-dcafc9f951db" (UID: "735cbee7-79d4-428a-baa7-dcafc9f951db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:24.553992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.553966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "735cbee7-79d4-428a-baa7-dcafc9f951db" (UID: "735cbee7-79d4-428a-baa7-dcafc9f951db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:24.555563 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.555539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "735cbee7-79d4-428a-baa7-dcafc9f951db" (UID: "735cbee7-79d4-428a-baa7-dcafc9f951db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:24.555667 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.555602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx" (OuterVolumeSpecName: "kube-api-access-8xwqx") pod "735cbee7-79d4-428a-baa7-dcafc9f951db" (UID: "735cbee7-79d4-428a-baa7-dcafc9f951db"). InnerVolumeSpecName "kube-api-access-8xwqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:24.564242 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.564209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "735cbee7-79d4-428a-baa7-dcafc9f951db" (UID: "735cbee7-79d4-428a-baa7-dcafc9f951db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:24.654632 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location\") pod \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") "
Sep 16 19:29:24.654829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654640    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqmqw\" (UniqueName: \"kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw\") pod \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") "
Sep 16 19:29:24.654829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config\") pod \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") "
Sep 16 19:29:24.654829 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls\") pod \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\" (UID: \"2f3331de-3f17-41f9-9ee7-4216a4e22da1\") "
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654941    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/735cbee7-79d4-428a-baa7-dcafc9f951db-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654961    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654974    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/735cbee7-79d4-428a-baa7-dcafc9f951db-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654992    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwqx\" (UniqueName: \"kubernetes.io/projected/735cbee7-79d4-428a-baa7-dcafc9f951db-kube-api-access-8xwqx\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.654984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f3331de-3f17-41f9-9ee7-4216a4e22da1" (UID: "2f3331de-3f17-41f9-9ee7-4216a4e22da1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:24.655057 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.655005    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/735cbee7-79d4-428a-baa7-dcafc9f951db-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.655331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.655097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f3331de-3f17-41f9-9ee7-4216a4e22da1" (UID: "2f3331de-3f17-41f9-9ee7-4216a4e22da1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:24.656883 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.656863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw" (OuterVolumeSpecName: "kube-api-access-gqmqw") pod "2f3331de-3f17-41f9-9ee7-4216a4e22da1" (UID: "2f3331de-3f17-41f9-9ee7-4216a4e22da1"). InnerVolumeSpecName "kube-api-access-gqmqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:24.656955 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.656922    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f3331de-3f17-41f9-9ee7-4216a4e22da1" (UID: "2f3331de-3f17-41f9-9ee7-4216a4e22da1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:24.755470 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.755443    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.755470 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.755468    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f3331de-3f17-41f9-9ee7-4216a4e22da1-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.755648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.755478    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:24.755648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:24.755488    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmqw\" (UniqueName: \"kubernetes.io/projected/2f3331de-3f17-41f9-9ee7-4216a4e22da1-kube-api-access-gqmqw\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:29:25.063072 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.062981    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerID="b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425" exitCode=0
Sep 16 19:29:25.063072 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.063039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"
Sep 16 19:29:25.063315 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.063071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerDied","Data":"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"}
Sep 16 19:29:25.063315 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.063127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq" event={"ID":"2f3331de-3f17-41f9-9ee7-4216a4e22da1","Type":"ContainerDied","Data":"943742ff4b55d3eec57e0e8dd0601ef1b053c7d408c23d5e5d7d08b3aba121db"}
Sep 16 19:29:25.063315 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.063156    2578 scope.go:117] "RemoveContainer" containerID="a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"
Sep 16 19:29:25.065328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.065303    2578 generic.go:356] "Generic (PLEG): container finished" podID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerID="a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2" exitCode=0
Sep 16 19:29:25.065448 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.065335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerDied","Data":"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"}
Sep 16 19:29:25.065448 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.065351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"
Sep 16 19:29:25.065448 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.065372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b" event={"ID":"735cbee7-79d4-428a-baa7-dcafc9f951db","Type":"ContainerDied","Data":"33213b6cbff6b3f848f6d56c75ad1f8ee87e52aebbd04c02207e361f39b38454"}
Sep 16 19:29:25.072730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.072711    2578 scope.go:117] "RemoveContainer" containerID="b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"
Sep 16 19:29:25.080515 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.080497    2578 scope.go:117] "RemoveContainer" containerID="367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"
Sep 16 19:29:25.087891 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.087870    2578 scope.go:117] "RemoveContainer" containerID="a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"
Sep 16 19:29:25.088220 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.088199    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146\": container with ID starting with a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146 not found: ID does not exist" containerID="a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"
Sep 16 19:29:25.088278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088229    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146"} err="failed to get container status \"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146\": rpc error: code = NotFound desc = could not find container \"a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146\": container with ID starting with a1b8b2b1330761cd69be10a7b63b41ed213b428cf5c7ff1e7dc608d210f59146 not found: ID does not exist"
Sep 16 19:29:25.088278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088245    2578 scope.go:117] "RemoveContainer" containerID="b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"
Sep 16 19:29:25.088509 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.088491    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425\": container with ID starting with b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425 not found: ID does not exist" containerID="b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"
Sep 16 19:29:25.088556 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088515    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425"} err="failed to get container status \"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425\": rpc error: code = NotFound desc = could not find container \"b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425\": container with ID starting with b7dc4283eab852fd85335ec6f00e458a68ab316371ae5e9256307d85e1fe8425 not found: ID does not exist"
Sep 16 19:29:25.088556 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088530    2578 scope.go:117] "RemoveContainer" containerID="367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"
Sep 16 19:29:25.088781 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.088761    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc\": container with ID starting with 367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc not found: ID does not exist" containerID="367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"
Sep 16 19:29:25.088884 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc"} err="failed to get container status \"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc\": rpc error: code = NotFound desc = could not find container \"367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc\": container with ID starting with 367b2a2b7542e158ff560a8822f887d06226eb15032f48a7149d5352f5e3b6dc not found: ID does not exist"
Sep 16 19:29:25.088884 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088811    2578 scope.go:117] "RemoveContainer" containerID="d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"
Sep 16 19:29:25.088884 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.088838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:29:25.093345 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.093325    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-776585ccf5-f7bgq"]
Sep 16 19:29:25.096972 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.096919    2578 scope.go:117] "RemoveContainer" containerID="a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"
Sep 16 19:29:25.107954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.107921    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:29:25.110063 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.110011    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-wgw6b"]
Sep 16 19:29:25.110121 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.110105    2578 scope.go:117] "RemoveContainer" containerID="0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"
Sep 16 19:29:25.117186 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.117170    2578 scope.go:117] "RemoveContainer" containerID="d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"
Sep 16 19:29:25.117461 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.117441    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138\": container with ID starting with d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138 not found: ID does not exist" containerID="d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"
Sep 16 19:29:25.117528 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.117468    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138"} err="failed to get container status \"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138\": rpc error: code = NotFound desc = could not find container \"d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138\": container with ID starting with d6af54e8a6d4976066ec6411b6e9d16960ce5048d6f094677fd91f97c45e2138 not found: ID does not exist"
Sep 16 19:29:25.117528 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.117486    2578 scope.go:117] "RemoveContainer" containerID="a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"
Sep 16 19:29:25.117782 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.117760    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2\": container with ID starting with a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2 not found: ID does not exist" containerID="a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"
Sep 16 19:29:25.117867 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.117796    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2"} err="failed to get container status \"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2\": rpc error: code = NotFound desc = could not find container \"a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2\": container with ID starting with a0934b2c8f966358b1f1930f730053da00ebbb3e12cb13461a89769be64f75e2 not found: ID does not exist"
Sep 16 19:29:25.117867 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.117818    2578 scope.go:117] "RemoveContainer" containerID="0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"
Sep 16 19:29:25.118122 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:29:25.118101    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50\": container with ID starting with 0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50 not found: ID does not exist" containerID="0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"
Sep 16 19:29:25.118163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.118127    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50"} err="failed to get container status \"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50\": rpc error: code = NotFound desc = could not find container \"0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50\": container with ID starting with 0d71f93ff6514e087e589434248a8a92a98507cf9af283b5e05c116592597b50 not found: ID does not exist"
Sep 16 19:29:25.603063 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.603034    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" path="/var/lib/kubelet/pods/2f3331de-3f17-41f9-9ee7-4216a4e22da1/volumes"
Sep 16 19:29:25.603518 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:25.603505    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" path="/var/lib/kubelet/pods/735cbee7-79d4-428a-baa7-dcafc9f951db/volumes"
Sep 16 19:29:26.981953 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.981915    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982207    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="storage-initializer"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982218    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="storage-initializer"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982232    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="storage-initializer"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982237    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="storage-initializer"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kube-rbac-proxy"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kube-rbac-proxy"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982259    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kube-rbac-proxy"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982264    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kube-rbac-proxy"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982290    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container"
Sep 16 19:29:26.982343 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982295    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container"
Sep 16 19:29:26.982736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982358    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kserve-container"
Sep 16 19:29:26.982736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kserve-container"
Sep 16 19:29:26.982736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982371    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3331de-3f17-41f9-9ee7-4216a4e22da1" containerName="kube-rbac-proxy"
Sep 16 19:29:26.982736 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.982378    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="735cbee7-79d4-428a-baa7-dcafc9f951db" containerName="kube-rbac-proxy"
Sep 16 19:29:26.985995 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.985977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:26.988914 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.988890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 16 19:29:26.989042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.988978    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:26.989042 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.989014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:26.989131 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.989019    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:29:26.990173 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.990150    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:26.990277 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.990207    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:29:26.990757 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:26.990741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:27.004591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.004570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:29:27.075394 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.075366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.075606 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.075403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkgh\" (UniqueName: \"kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.075606 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.075423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.075606 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.075458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.075606 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.075553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.176880 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.176821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177076 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.176891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177076 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.176931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkgh\" (UniqueName: \"kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177076 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.176954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177076 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.176995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177447 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.177427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177730 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.177708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.177778 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.177708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.179405 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.179388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.187499 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.187466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkgh\" (UniqueName: \"kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kcr96\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.296282 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.296198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:27.569991 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:27.569965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:29:27.570743 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:29:27.570715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd4c9bd_67d0_4f0c_afb0_edb66d414fee.slice/crio-ece9b3a6fec7c4f8b02aaa17788217d52c1035d117042a3b8cbb919301454acb WatchSource:0}: Error finding container ece9b3a6fec7c4f8b02aaa17788217d52c1035d117042a3b8cbb919301454acb: Status 404 returned error can't find the container with id ece9b3a6fec7c4f8b02aaa17788217d52c1035d117042a3b8cbb919301454acb
Sep 16 19:29:28.082441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:28.082402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerStarted","Data":"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"}
Sep 16 19:29:28.082943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:28.082476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerStarted","Data":"ece9b3a6fec7c4f8b02aaa17788217d52c1035d117042a3b8cbb919301454acb"}
Sep 16 19:29:33.107201 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:33.107164    2578 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerID="1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1" exitCode=0
Sep 16 19:29:33.107710 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:33.107241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerDied","Data":"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"}
Sep 16 19:29:34.113334 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.113298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerStarted","Data":"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"}
Sep 16 19:29:34.113717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.113363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerStarted","Data":"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"}
Sep 16 19:29:34.113717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.113617    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:34.113717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.113671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:34.115084 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.115057    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:29:34.142639 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:34.142591    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podStartSLOduration=3.5877432000000002 podStartE2EDuration="8.142576093s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.554832893s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:34.139992934 +0000 UTC m=+2633.111470188" watchObservedRunningTime="2026-09-16 19:29:34.142576093 +0000 UTC m=+2633.114053343"
Sep 16 19:29:35.118290 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:35.118254    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:29:40.122467 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:40.122434    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:29:40.123101 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:40.123070    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:29:50.123424 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:29:50.123375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:30:00.123053 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:00.123010    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:30:10.123750 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:10.123712    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:30:20.123741 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:20.123710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:30:28.080446 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:28.080412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:30:28.080931 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:28.080748    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" containerID="cri-o://63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42" gracePeriod=30
Sep 16 19:30:28.080931 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:28.080820    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kube-rbac-proxy" containerID="cri-o://fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89" gracePeriod=30
Sep 16 19:30:28.337249 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:28.337214    2578 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerID="fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89" exitCode=2
Sep 16 19:30:28.337409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:28.337288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerDied","Data":"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"}
Sep 16 19:30:30.118981 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.118937    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:30:30.123256 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.123222    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:30:30.819275 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.819249    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:30:30.913286 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location\") pod \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") "
Sep 16 19:30:30.913286 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert\") pod \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") "
Sep 16 19:30:30.913286 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config\") pod \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") "
Sep 16 19:30:30.913540 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffkgh\" (UniqueName: \"kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh\") pod \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") "
Sep 16 19:30:30.913540 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls\") pod \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\" (UID: \"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee\") "
Sep 16 19:30:30.913655 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" (UID: "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:30.913655 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.913599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" (UID: "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:30.915589 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.915553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" (UID: "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:30.915776 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.915557    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh" (OuterVolumeSpecName: "kube-api-access-ffkgh") pod "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" (UID: "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee"). InnerVolumeSpecName "kube-api-access-ffkgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:30.923474 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:30.923451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" (UID: "5dd4c9bd-67d0-4f0c-afb0-edb66d414fee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:31.013943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.013907    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:30:31.013943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.013938    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:30:31.013943 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.013948    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkgh\" (UniqueName: \"kubernetes.io/projected/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kube-api-access-ffkgh\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:30:31.014127 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.013958    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:30:31.014127 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.013968    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:30:31.352739 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.352705    2578 generic.go:356] "Generic (PLEG): container finished" podID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerID="63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42" exitCode=0
Sep 16 19:30:31.353132 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.352767    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"
Sep 16 19:30:31.353132 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.352786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerDied","Data":"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"}
Sep 16 19:30:31.353132 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.352891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96" event={"ID":"5dd4c9bd-67d0-4f0c-afb0-edb66d414fee","Type":"ContainerDied","Data":"ece9b3a6fec7c4f8b02aaa17788217d52c1035d117042a3b8cbb919301454acb"}
Sep 16 19:30:31.353132 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.352918    2578 scope.go:117] "RemoveContainer" containerID="fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"
Sep 16 19:30:31.361366 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.361349    2578 scope.go:117] "RemoveContainer" containerID="63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"
Sep 16 19:30:31.368749 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.368732    2578 scope.go:117] "RemoveContainer" containerID="1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"
Sep 16 19:30:31.375709 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.375687    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:30:31.376353 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.376335    2578 scope.go:117] "RemoveContainer" containerID="fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"
Sep 16 19:30:31.376648 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:30:31.376619    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89\": container with ID starting with fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89 not found: ID does not exist" containerID="fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"
Sep 16 19:30:31.376769 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.376654    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89"} err="failed to get container status \"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89\": rpc error: code = NotFound desc = could not find container \"fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89\": container with ID starting with fccbdf6168c715bd39cc19659b2154776a3d156442819095c247381630bdcb89 not found: ID does not exist"
Sep 16 19:30:31.376769 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.376675    2578 scope.go:117] "RemoveContainer" containerID="63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"
Sep 16 19:30:31.377022 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:30:31.376996    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42\": container with ID starting with 63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42 not found: ID does not exist" containerID="63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"
Sep 16 19:30:31.377064 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.377026    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42"} err="failed to get container status \"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42\": rpc error: code = NotFound desc = could not find container \"63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42\": container with ID starting with 63a379d173cc7cbd1c37e5e289964ceb32aef29e8cee37ab82e58508597cfc42 not found: ID does not exist"
Sep 16 19:30:31.377064 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.377040    2578 scope.go:117] "RemoveContainer" containerID="1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"
Sep 16 19:30:31.377308 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:30:31.377291    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1\": container with ID starting with 1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1 not found: ID does not exist" containerID="1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"
Sep 16 19:30:31.377364 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.377310    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1"} err="failed to get container status \"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1\": rpc error: code = NotFound desc = could not find container \"1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1\": container with ID starting with 1974f375298d776682bd6d76f6cd78e04bca2e22cd07b5231194db95c3064ef1 not found: ID does not exist"
Sep 16 19:30:31.380371 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.380348    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kcr96"]
Sep 16 19:30:31.602921 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:31.602800    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" path="/var/lib/kubelet/pods/5dd4c9bd-67d0-4f0c-afb0-edb66d414fee/volumes"
Sep 16 19:30:33.296727 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296650    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296959    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296970    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296985    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="storage-initializer"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296990    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="storage-initializer"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.296997    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kube-rbac-proxy"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.297003    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kube-rbac-proxy"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.297056    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kube-rbac-proxy"
Sep 16 19:30:33.297085 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.297063    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4c9bd-67d0-4f0c-afb0-edb66d414fee" containerName="kserve-container"
Sep 16 19:30:33.299987 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.299955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.303012 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.302981    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:33.303147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.302987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:30:33.303147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.303080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:30:33.303147 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.302986    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 16 19:30:33.303328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.302989    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:33.304268 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.304248    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:30:33.304268 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.304264    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:30:33.315734 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.315708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:30:33.434513 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.434475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.434690 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.434541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.434690 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.434577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.434690 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.434616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.434690 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.434642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6nm\" (UniqueName: \"kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6nm\" (UniqueName: \"kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536287 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536582 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.536954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.536843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.538602 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.538585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.545806 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.545780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6nm\" (UniqueName: \"kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.610013 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.609986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:33.882070 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:33.882032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:30:33.884326 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:30:33.884297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2461bc43_c95f_4b26_9012_fb9e3cbc7879.slice/crio-59e53b3e341c8b8f8b37ab467b863c7de5bbecabef4a80feae55af1c9d5b6cd9 WatchSource:0}: Error finding container 59e53b3e341c8b8f8b37ab467b863c7de5bbecabef4a80feae55af1c9d5b6cd9: Status 404 returned error can't find the container with id 59e53b3e341c8b8f8b37ab467b863c7de5bbecabef4a80feae55af1c9d5b6cd9
Sep 16 19:30:34.367637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:34.367601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerStarted","Data":"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"}
Sep 16 19:30:34.368017 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:34.367663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerStarted","Data":"59e53b3e341c8b8f8b37ab467b863c7de5bbecabef4a80feae55af1c9d5b6cd9"}
Sep 16 19:30:38.385129 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:38.385041    2578 generic.go:356] "Generic (PLEG): container finished" podID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerID="ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6" exitCode=0
Sep 16 19:30:38.385129 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:38.385116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerDied","Data":"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"}
Sep 16 19:30:39.391869 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:39.391825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerStarted","Data":"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"}
Sep 16 19:30:39.392370 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:39.391925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerStarted","Data":"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"}
Sep 16 19:30:39.392370 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:39.392148    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:39.416289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:39.416236    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podStartSLOduration=2.264225763 podStartE2EDuration="6.416215454s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.151989691s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:39.413952689 +0000 UTC m=+2698.385429942" watchObservedRunningTime="2026-09-16 19:30:39.416215454 +0000 UTC m=+2698.387692703"
Sep 16 19:30:40.395657 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:40.395624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:40.396962 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:40.396934    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:30:41.399648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:41.399605    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:30:46.404232 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:46.404203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:30:46.404795 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:46.404765    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:30:56.405379 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:30:56.405339    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:31:02.083216 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.083129    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:31:02.086705 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.086683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.089454 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.089434    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:31:02.089702 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.089686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 16 19:31:02.089771 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.089711    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:31:02.096935 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.096912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:31:02.166909 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.166872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.167090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.166931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjkgr\" (UniqueName: \"kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.167090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.166959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.167090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.166990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.167090 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.167052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268382 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjkgr\" (UniqueName: \"kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268382 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268631 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.268631 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:31:02.268592    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 16 19:31:02.268810 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:31:02.268681    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls podName:5038e05e-39c3-479b-82e0-c06683e0de53 nodeName:}" failed. No retries permitted until 2026-09-16 19:31:02.76866111 +0000 UTC m=+2721.740138357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 16 19:31:02.268810 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.268748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.269136 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.269118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.269177 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.269144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.281330 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.281303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjkgr\" (UniqueName: \"kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.772126 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.772084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.774675 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.774653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:02.997275 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:02.997241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:03.272965 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:03.272935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:31:03.275009 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:31:03.274978    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5038e05e_39c3_479b_82e0_c06683e0de53.slice/crio-d43b921b3052f452daa0962c06c0dbab2ad5ff1d3aaef530142fdfa1d19fcb96 WatchSource:0}: Error finding container d43b921b3052f452daa0962c06c0dbab2ad5ff1d3aaef530142fdfa1d19fcb96: Status 404 returned error can't find the container with id d43b921b3052f452daa0962c06c0dbab2ad5ff1d3aaef530142fdfa1d19fcb96
Sep 16 19:31:03.493783 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:03.493742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerStarted","Data":"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"}
Sep 16 19:31:03.493986 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:03.493827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerStarted","Data":"d43b921b3052f452daa0962c06c0dbab2ad5ff1d3aaef530142fdfa1d19fcb96"}
Sep 16 19:31:04.500146 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:04.500112    2578 generic.go:356] "Generic (PLEG): container finished" podID="5038e05e-39c3-479b-82e0-c06683e0de53" containerID="2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945" exitCode=0
Sep 16 19:31:04.500543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:04.500196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerDied","Data":"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"}
Sep 16 19:31:05.507299 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.507259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerStarted","Data":"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"}
Sep 16 19:31:05.507787 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.507314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerStarted","Data":"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"}
Sep 16 19:31:05.507787 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.507540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:05.507787 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.507648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:05.508924 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.508901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:05.535173 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:05.535112    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podStartSLOduration=2.392981879 podStartE2EDuration="3.535092336s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.142110457s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:05.531775773 +0000 UTC m=+2724.503253025" watchObservedRunningTime="2026-09-16 19:31:05.535092336 +0000 UTC m=+2724.506569586"
Sep 16 19:31:06.404953 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:06.404912    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:31:06.512420 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:06.512376    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:11.517842 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:11.517813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:31:11.518349 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:11.518324    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:16.404897 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:16.404834    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:31:21.519092 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:21.519047    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:26.406039 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:26.406004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:31:31.519028 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:31.518983    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:34.671294 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:34.671260    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:31:34.671785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:34.671564    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" containerID="cri-o://9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad" gracePeriod=30
Sep 16 19:31:34.671785 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:34.671615    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kube-rbac-proxy" containerID="cri-o://6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a" gracePeriod=30
Sep 16 19:31:35.630733 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:35.630693    2578 generic.go:356] "Generic (PLEG): container finished" podID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerID="6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a" exitCode=2
Sep 16 19:31:35.630933 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:35.630767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerDied","Data":"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"}
Sep 16 19:31:36.400058 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:36.400016    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:31:36.405358 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:36.405332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:31:37.423463 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.423436    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:31:37.566617 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location\") pod \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") "
Sep 16 19:31:37.566791 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config\") pod \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") "
Sep 16 19:31:37.566791 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert\") pod \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") "
Sep 16 19:31:37.566791 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr6nm\" (UniqueName: \"kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm\") pod \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") "
Sep 16 19:31:37.567009 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls\") pod \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\" (UID: \"2461bc43-c95f-4b26-9012-fb9e3cbc7879\") "
Sep 16 19:31:37.567009 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.566945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2461bc43-c95f-4b26-9012-fb9e3cbc7879" (UID: "2461bc43-c95f-4b26-9012-fb9e3cbc7879"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:37.567146 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.567118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2461bc43-c95f-4b26-9012-fb9e3cbc7879" (UID: "2461bc43-c95f-4b26-9012-fb9e3cbc7879"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:37.569048 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.569024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2461bc43-c95f-4b26-9012-fb9e3cbc7879" (UID: "2461bc43-c95f-4b26-9012-fb9e3cbc7879"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:37.569169 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.569040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm" (OuterVolumeSpecName: "kube-api-access-lr6nm") pod "2461bc43-c95f-4b26-9012-fb9e3cbc7879" (UID: "2461bc43-c95f-4b26-9012-fb9e3cbc7879"). InnerVolumeSpecName "kube-api-access-lr6nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:37.576824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.576794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2461bc43-c95f-4b26-9012-fb9e3cbc7879" (UID: "2461bc43-c95f-4b26-9012-fb9e3cbc7879"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:37.641603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.641566    2578 generic.go:356] "Generic (PLEG): container finished" podID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerID="9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad" exitCode=0
Sep 16 19:31:37.641753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.641622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"
Sep 16 19:31:37.641753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.641631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerDied","Data":"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"}
Sep 16 19:31:37.641753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.641689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm" event={"ID":"2461bc43-c95f-4b26-9012-fb9e3cbc7879","Type":"ContainerDied","Data":"59e53b3e341c8b8f8b37ab467b863c7de5bbecabef4a80feae55af1c9d5b6cd9"}
Sep 16 19:31:37.641753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.641716    2578 scope.go:117] "RemoveContainer" containerID="6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"
Sep 16 19:31:37.650184 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.650166    2578 scope.go:117] "RemoveContainer" containerID="9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"
Sep 16 19:31:37.657542 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.657524    2578 scope.go:117] "RemoveContainer" containerID="ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"
Sep 16 19:31:37.661210 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.661189    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:31:37.666128 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.665578    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4rsvm"]
Sep 16 19:31:37.667569 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667553    2578 scope.go:117] "RemoveContainer" containerID="6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"
Sep 16 19:31:37.667875 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667830    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.667963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667876    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2461bc43-c95f-4b26-9012-fb9e3cbc7879-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.667963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667893    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6nm\" (UniqueName: \"kubernetes.io/projected/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kube-api-access-lr6nm\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.667963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667908    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2461bc43-c95f-4b26-9012-fb9e3cbc7879-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.667963 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667921    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2461bc43-c95f-4b26-9012-fb9e3cbc7879-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.667963 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:31:37.667933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a\": container with ID starting with 6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a not found: ID does not exist" containerID="6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"
Sep 16 19:31:37.668155 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667963    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a"} err="failed to get container status \"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a\": rpc error: code = NotFound desc = could not find container \"6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a\": container with ID starting with 6ba356ac9ba655d8cadc982c0fdd40fd6454c43f1e04dd5432b9375e51178d7a not found: ID does not exist"
Sep 16 19:31:37.668155 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.667985    2578 scope.go:117] "RemoveContainer" containerID="9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"
Sep 16 19:31:37.668282 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:31:37.668262    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad\": container with ID starting with 9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad not found: ID does not exist" containerID="9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"
Sep 16 19:31:37.668325 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.668297    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad"} err="failed to get container status \"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad\": rpc error: code = NotFound desc = could not find container \"9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad\": container with ID starting with 9e73a2661d04d74821b81f7386dd81eb92330952df404eea176d631e438f82ad not found: ID does not exist"
Sep 16 19:31:37.668325 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.668317    2578 scope.go:117] "RemoveContainer" containerID="ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"
Sep 16 19:31:37.668608 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:31:37.668586    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6\": container with ID starting with ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6 not found: ID does not exist" containerID="ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"
Sep 16 19:31:37.668702 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:37.668613    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6"} err="failed to get container status \"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6\": rpc error: code = NotFound desc = could not find container \"ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6\": container with ID starting with ae1e8f770a0eecbcd5fd70cf13aaacb1e773b8deb4fd011721ade73be8b6d5f6 not found: ID does not exist"
Sep 16 19:31:39.603700 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:39.603668    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" path="/var/lib/kubelet/pods/2461bc43-c95f-4b26-9012-fb9e3cbc7879/volumes"
Sep 16 19:31:41.518272 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:41.518236    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:31:51.519070 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:31:51.519024    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:32:01.518663 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:01.518621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:32:11.518761 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:11.518664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:32:17.602565 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:17.602537    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:32:22.134949 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:22.134909    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:32:22.135449 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:22.135141    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container" containerID="cri-o://45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461" gracePeriod=30
Sep 16 19:32:22.135449 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:22.135195    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kube-rbac-proxy" containerID="cri-o://aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2" gracePeriod=30
Sep 16 19:32:22.824058 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:22.824026    2578 generic.go:356] "Generic (PLEG): container finished" podID="5038e05e-39c3-479b-82e0-c06683e0de53" containerID="aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2" exitCode=2
Sep 16 19:32:22.824246 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:22.824079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerDied","Data":"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"}
Sep 16 19:32:23.184075 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184040    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184381    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kube-rbac-proxy"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kube-rbac-proxy"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184404    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="storage-initializer"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184410    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="storage-initializer"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184460    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kube-rbac-proxy"
Sep 16 19:32:23.184487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.184471    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2461bc43-c95f-4b26-9012-fb9e3cbc7879" containerName="kserve-container"
Sep 16 19:32:23.187600 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.187583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.190672 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.190654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:32:23.190882 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.190864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 19:32:23.199903 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.199866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:23.332056 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.332015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.332223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.332108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9fq\" (UniqueName: \"kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.332223 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.332144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.332304 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.332239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.433340 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.433303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9fq\" (UniqueName: \"kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.433340 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.433343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.433566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.433396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.433566 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.433428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.433893 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.433840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.434172 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.434129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.436148 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.436125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.443270 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.443225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9fq\" (UniqueName: \"kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.497926 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.497900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:23.772275 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.772149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:23.774938 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:32:23.774911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bf60d0_9062_4c98_a416_accc9a683519.slice/crio-7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769 WatchSource:0}: Error finding container 7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769: Status 404 returned error can't find the container with id 7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769
Sep 16 19:32:23.829840 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:23.829811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerStarted","Data":"7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769"}
Sep 16 19:32:24.836244 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:24.836212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerStarted","Data":"5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"}
Sep 16 19:32:26.512902 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.512831    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 16 19:32:26.682232 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.682206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:32:26.762767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.762732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") pod \"5038e05e-39c3-479b-82e0-c06683e0de53\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") "
Sep 16 19:32:26.762945 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.762776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location\") pod \"5038e05e-39c3-479b-82e0-c06683e0de53\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") "
Sep 16 19:32:26.762945 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.762800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjkgr\" (UniqueName: \"kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr\") pod \"5038e05e-39c3-479b-82e0-c06683e0de53\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") "
Sep 16 19:32:26.762945 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.762893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config\") pod \"5038e05e-39c3-479b-82e0-c06683e0de53\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") "
Sep 16 19:32:26.762945 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.762919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert\") pod \"5038e05e-39c3-479b-82e0-c06683e0de53\" (UID: \"5038e05e-39c3-479b-82e0-c06683e0de53\") "
Sep 16 19:32:26.763180 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.763082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5038e05e-39c3-479b-82e0-c06683e0de53" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:26.763301 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.763278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5038e05e-39c3-479b-82e0-c06683e0de53" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:26.763356 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.763308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5038e05e-39c3-479b-82e0-c06683e0de53" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:26.764771 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.764748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5038e05e-39c3-479b-82e0-c06683e0de53" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:26.764872 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.764834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr" (OuterVolumeSpecName: "kube-api-access-sjkgr") pod "5038e05e-39c3-479b-82e0-c06683e0de53" (UID: "5038e05e-39c3-479b-82e0-c06683e0de53"). InnerVolumeSpecName "kube-api-access-sjkgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:26.847801 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.847774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/0.log"
Sep 16 19:32:26.847974 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.847817    2578 generic.go:356] "Generic (PLEG): container finished" podID="74bf60d0-9062-4c98-a416-accc9a683519" containerID="5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4" exitCode=1
Sep 16 19:32:26.847974 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.847895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerDied","Data":"5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"}
Sep 16 19:32:26.850161 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.850138    2578 generic.go:356] "Generic (PLEG): container finished" podID="5038e05e-39c3-479b-82e0-c06683e0de53" containerID="45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461" exitCode=0
Sep 16 19:32:26.850258 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.850190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"
Sep 16 19:32:26.850258 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.850216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerDied","Data":"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"}
Sep 16 19:32:26.850331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.850275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48" event={"ID":"5038e05e-39c3-479b-82e0-c06683e0de53","Type":"ContainerDied","Data":"d43b921b3052f452daa0962c06c0dbab2ad5ff1d3aaef530142fdfa1d19fcb96"}
Sep 16 19:32:26.850331 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.850312    2578 scope.go:117] "RemoveContainer" containerID="aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"
Sep 16 19:32:26.860271 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.860248    2578 scope.go:117] "RemoveContainer" containerID="45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"
Sep 16 19:32:26.863731 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.863709    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:26.863798 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.863740    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5038e05e-39c3-479b-82e0-c06683e0de53-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:26.863798 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.863757    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5038e05e-39c3-479b-82e0-c06683e0de53-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:26.863798 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.863771    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5038e05e-39c3-479b-82e0-c06683e0de53-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:26.863798 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.863785    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjkgr\" (UniqueName: \"kubernetes.io/projected/5038e05e-39c3-479b-82e0-c06683e0de53-kube-api-access-sjkgr\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:26.868772 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.868751    2578 scope.go:117] "RemoveContainer" containerID="2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"
Sep 16 19:32:26.878717 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.878701    2578 scope.go:117] "RemoveContainer" containerID="aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"
Sep 16 19:32:26.879013 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:32:26.878993    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2\": container with ID starting with aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2 not found: ID does not exist" containerID="aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"
Sep 16 19:32:26.879171 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.879020    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2"} err="failed to get container status \"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2\": rpc error: code = NotFound desc = could not find container \"aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2\": container with ID starting with aa7477e130b1d732d6ea07731b1847fc1719be4adf330cf268eea2a38a85cfe2 not found: ID does not exist"
Sep 16 19:32:26.879171 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.879037    2578 scope.go:117] "RemoveContainer" containerID="45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"
Sep 16 19:32:26.879327 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:32:26.879296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461\": container with ID starting with 45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461 not found: ID does not exist" containerID="45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"
Sep 16 19:32:26.879414 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.879330    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461"} err="failed to get container status \"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461\": rpc error: code = NotFound desc = could not find container \"45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461\": container with ID starting with 45be460edc68a0b12281f951a2840b8f080e6bc9dccc06842da090cb23d82461 not found: ID does not exist"
Sep 16 19:32:26.879514 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.879499    2578 scope.go:117] "RemoveContainer" containerID="2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"
Sep 16 19:32:26.879956 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:32:26.879907    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945\": container with ID starting with 2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945 not found: ID does not exist" containerID="2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"
Sep 16 19:32:26.879956 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.879933    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945"} err="failed to get container status \"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945\": rpc error: code = NotFound desc = could not find container \"2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945\": container with ID starting with 2e0f7aef075c94fbb31fc91319284f774c9c1df7a95ac862efd4cc3c85a92945 not found: ID does not exist"
Sep 16 19:32:26.881788 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.881770    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:32:26.886777 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:26.886758    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5954896d5f-4qq48"]
Sep 16 19:32:27.603703 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:27.603670    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" path="/var/lib/kubelet/pods/5038e05e-39c3-479b-82e0-c06683e0de53/volumes"
Sep 16 19:32:27.857017 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:27.856929    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/0.log"
Sep 16 19:32:27.857183 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:27.857020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerStarted","Data":"37b63602b75f05fe9af90a27a94c3a3d52a971870858129ca355ea596d8716ff"}
Sep 16 19:32:29.868233 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.868205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/1.log"
Sep 16 19:32:29.868612 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.868597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/0.log"
Sep 16 19:32:29.868654 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.868629    2578 generic.go:356] "Generic (PLEG): container finished" podID="74bf60d0-9062-4c98-a416-accc9a683519" containerID="37b63602b75f05fe9af90a27a94c3a3d52a971870858129ca355ea596d8716ff" exitCode=1
Sep 16 19:32:29.868694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.868677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerDied","Data":"37b63602b75f05fe9af90a27a94c3a3d52a971870858129ca355ea596d8716ff"}
Sep 16 19:32:29.868747 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.868729    2578 scope.go:117] "RemoveContainer" containerID="5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"
Sep 16 19:32:29.869095 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.869077    2578 scope.go:117] "RemoveContainer" containerID="5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"
Sep 16 19:32:29.879254 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:32:29.879230    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_kserve-ci-e2e-test(74bf60d0-9062-4c98-a416-accc9a683519)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" podUID="74bf60d0-9062-4c98-a416-accc9a683519"
Sep 16 19:32:29.879305 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:32:29.879247    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_kserve-ci-e2e-test_74bf60d0-9062-4c98-a416-accc9a683519_0 in pod sandbox 7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769 from index: no such id: '5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4'" containerID="5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"
Sep 16 19:32:29.879305 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:29.879280    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_kserve-ci-e2e-test_74bf60d0-9062-4c98-a416-accc9a683519_0 in pod sandbox 7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769 from index: no such id: '5a45e52ab67fab260906da3ad4213b9b3dc44a7af34bcb7c0ca539d5e2435af4'"
Sep 16 19:32:30.874292 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:30.874262    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/1.log"
Sep 16 19:32:33.143680 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.143645    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:33.265687 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.265664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/1.log"
Sep 16 19:32:33.267987 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.267971    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:33.417920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.417813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls\") pod \"74bf60d0-9062-4c98-a416-accc9a683519\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") "
Sep 16 19:32:33.417920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.417903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location\") pod \"74bf60d0-9062-4c98-a416-accc9a683519\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") "
Sep 16 19:32:33.418106 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.418072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr9fq\" (UniqueName: \"kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq\") pod \"74bf60d0-9062-4c98-a416-accc9a683519\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") "
Sep 16 19:32:33.418153 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.418116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config\") pod \"74bf60d0-9062-4c98-a416-accc9a683519\" (UID: \"74bf60d0-9062-4c98-a416-accc9a683519\") "
Sep 16 19:32:33.418153 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.418131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74bf60d0-9062-4c98-a416-accc9a683519" (UID: "74bf60d0-9062-4c98-a416-accc9a683519"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:33.418382 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.418361    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74bf60d0-9062-4c98-a416-accc9a683519-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:33.418511 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.418483    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74bf60d0-9062-4c98-a416-accc9a683519" (UID: "74bf60d0-9062-4c98-a416-accc9a683519"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:33.420146 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.420121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74bf60d0-9062-4c98-a416-accc9a683519" (UID: "74bf60d0-9062-4c98-a416-accc9a683519"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:33.420246 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.420231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq" (OuterVolumeSpecName: "kube-api-access-dr9fq") pod "74bf60d0-9062-4c98-a416-accc9a683519" (UID: "74bf60d0-9062-4c98-a416-accc9a683519"). InnerVolumeSpecName "kube-api-access-dr9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:33.518761 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.518713    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74bf60d0-9062-4c98-a416-accc9a683519-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:33.518761 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.518757    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9fq\" (UniqueName: \"kubernetes.io/projected/74bf60d0-9062-4c98-a416-accc9a683519-kube-api-access-dr9fq\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:33.518761 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.518771    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74bf60d0-9062-4c98-a416-accc9a683519-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:32:33.887443 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.887416    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb_74bf60d0-9062-4c98-a416-accc9a683519/storage-initializer/1.log"
Sep 16 19:32:33.887618 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.887490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb" event={"ID":"74bf60d0-9062-4c98-a416-accc9a683519","Type":"ContainerDied","Data":"7a8567559c49b63e4bc260d6ca82d02eb49357f7f41a4faf9d19122b8c00a769"}
Sep 16 19:32:33.887618 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.887515    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"
Sep 16 19:32:33.887618 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.887542    2578 scope.go:117] "RemoveContainer" containerID="37b63602b75f05fe9af90a27a94c3a3d52a971870858129ca355ea596d8716ff"
Sep 16 19:32:33.944947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.944896    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:33.947178 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:33.947152    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5df9f8bfc5-fcfjb"]
Sep 16 19:32:34.350947 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.350921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351213    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351223    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351236    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351269    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351274    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="storage-initializer"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kube-rbac-proxy"
Sep 16 19:32:34.351318 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351284    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kube-rbac-proxy"
Sep 16 19:32:34.351637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351331    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kube-rbac-proxy"
Sep 16 19:32:34.351637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351340    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.351637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351347    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5038e05e-39c3-479b-82e0-c06683e0de53" containerName="kserve-container"
Sep 16 19:32:34.351637 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.351353    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bf60d0-9062-4c98-a416-accc9a683519" containerName="storage-initializer"
Sep 16 19:32:34.355648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.355630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.359032 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359011    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:32:34.359307 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:32:34.359506 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:32:34.359647 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359625    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:34.359701 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:32:34.359743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.359634    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 19:32:34.360175 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.360161    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:32:34.368077 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.368056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:32:34.426403 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.426370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgz6\" (UniqueName: \"kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.426544 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.426413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.426544 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.426497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.426681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.426562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.426681 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.426607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.527801 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.527769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.527960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.527812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.527960 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.527843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgz6\" (UniqueName: \"kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.528078 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.528052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.528190 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.528166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.528470 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.528446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.528578 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.528553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.528687 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.528670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.530409 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.530390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.536986 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.536969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgz6\" (UniqueName: \"kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.666045 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.665960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:34.940052 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:34.940025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:32:34.942175 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:32:34.942147    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5555c3f6_1bf0_439c_9ba9_384318215caf.slice/crio-000538556d7723e68ea587d9de6d291385d66869322ae11905b4802328d031bc WatchSource:0}: Error finding container 000538556d7723e68ea587d9de6d291385d66869322ae11905b4802328d031bc: Status 404 returned error can't find the container with id 000538556d7723e68ea587d9de6d291385d66869322ae11905b4802328d031bc
Sep 16 19:32:35.602601 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:35.602558    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bf60d0-9062-4c98-a416-accc9a683519" path="/var/lib/kubelet/pods/74bf60d0-9062-4c98-a416-accc9a683519/volumes"
Sep 16 19:32:35.898587 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:35.898502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerStarted","Data":"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"}
Sep 16 19:32:35.898587 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:35.898563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerStarted","Data":"000538556d7723e68ea587d9de6d291385d66869322ae11905b4802328d031bc"}
Sep 16 19:32:36.904212 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:36.904173    2578 generic.go:356] "Generic (PLEG): container finished" podID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerID="e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c" exitCode=0
Sep 16 19:32:36.904591 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:36.904229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerDied","Data":"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"}
Sep 16 19:32:37.911020 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.910987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerStarted","Data":"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"}
Sep 16 19:32:37.911376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.911050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerStarted","Data":"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"}
Sep 16 19:32:37.911376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.911084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:37.911376 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.911149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:37.912428 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.912407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:32:37.931753 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:37.931697    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podStartSLOduration=2.832099917 podStartE2EDuration="3.931684845s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.099584928s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:37.931323522 +0000 UTC m=+2816.902800771" watchObservedRunningTime="2026-09-16 19:32:37.931684845 +0000 UTC m=+2816.903162094"
Sep 16 19:32:38.915567 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:38.915523    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:32:43.920317 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:43.920288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:32:43.921009 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:43.920978    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:32:53.920914 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:32:53.920831    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:03.921380 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:03.921341    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:13.921284 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:13.921222    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:23.921415 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:23.921379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:33.921499 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:33.921409    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:43.921609 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:43.921566    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:33:53.922303 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:53.922274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:33:54.303675 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:54.303580    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:33:54.303897 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:54.303867    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container" containerID="cri-o://507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc" gracePeriod=30
Sep 16 19:33:54.303992 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:54.303917    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kube-rbac-proxy" containerID="cri-o://1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33" gracePeriod=30
Sep 16 19:33:55.233254 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.233219    2578 generic.go:356] "Generic (PLEG): container finished" podID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerID="1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33" exitCode=2
Sep 16 19:33:55.233724 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.233291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerDied","Data":"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"}
Sep 16 19:33:55.399550 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.399511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:33:55.403087 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.403065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.405892 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.405840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 19:33:55.406015 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.405932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:33:55.414360 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.414336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:33:55.589491 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.589458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.589648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.589502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.589648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.589550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.589648 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.589602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.690667 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.690628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.690873 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.690674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.690873 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.690719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.690873 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.690754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.691122 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.691091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.691565 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.691498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.693447 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.693420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.700231 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.700205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.714091 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.714067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:33:55.984390 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:55.984366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:33:55.986805 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:33:55.986777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d1a5f3_e425_481a_b007_0abb34b5829e.slice/crio-4068ca2ce09ac7bb86aeeabe4445486aec485a6bd383bbf6f7ba6208945496c3 WatchSource:0}: Error finding container 4068ca2ce09ac7bb86aeeabe4445486aec485a6bd383bbf6f7ba6208945496c3: Status 404 returned error can't find the container with id 4068ca2ce09ac7bb86aeeabe4445486aec485a6bd383bbf6f7ba6208945496c3
Sep 16 19:33:56.240442 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:56.240351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerStarted","Data":"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"}
Sep 16 19:33:56.240442 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:56.240428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerStarted","Data":"4068ca2ce09ac7bb86aeeabe4445486aec485a6bd383bbf6f7ba6208945496c3"}
Sep 16 19:33:58.916510 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:58.916470    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 16 19:33:59.051799 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.051774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:33:59.223014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.222924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config\") pod \"5555c3f6-1bf0-439c-9ba9-384318215caf\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") "
Sep 16 19:33:59.223014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.222967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtgz6\" (UniqueName: \"kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6\") pod \"5555c3f6-1bf0-439c-9ba9-384318215caf\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") "
Sep 16 19:33:59.223014 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.222985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert\") pod \"5555c3f6-1bf0-439c-9ba9-384318215caf\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") "
Sep 16 19:33:59.223312 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.223045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location\") pod \"5555c3f6-1bf0-439c-9ba9-384318215caf\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") "
Sep 16 19:33:59.223312 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.223067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls\") pod \"5555c3f6-1bf0-439c-9ba9-384318215caf\" (UID: \"5555c3f6-1bf0-439c-9ba9-384318215caf\") "
Sep 16 19:33:59.223441 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.223413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5555c3f6-1bf0-439c-9ba9-384318215caf" (UID: "5555c3f6-1bf0-439c-9ba9-384318215caf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:59.223497 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.223430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5555c3f6-1bf0-439c-9ba9-384318215caf" (UID: "5555c3f6-1bf0-439c-9ba9-384318215caf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:59.223497 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.223420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5555c3f6-1bf0-439c-9ba9-384318215caf" (UID: "5555c3f6-1bf0-439c-9ba9-384318215caf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:59.225363 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.225339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5555c3f6-1bf0-439c-9ba9-384318215caf" (UID: "5555c3f6-1bf0-439c-9ba9-384318215caf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:59.225487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.225378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6" (OuterVolumeSpecName: "kube-api-access-xtgz6") pod "5555c3f6-1bf0-439c-9ba9-384318215caf" (UID: "5555c3f6-1bf0-439c-9ba9-384318215caf"). InnerVolumeSpecName "kube-api-access-xtgz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:59.255034 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.255003    2578 generic.go:356] "Generic (PLEG): container finished" podID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerID="507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc" exitCode=0
Sep 16 19:33:59.255152 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.255062    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"
Sep 16 19:33:59.255152 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.255094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerDied","Data":"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"}
Sep 16 19:33:59.255231 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.255162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr" event={"ID":"5555c3f6-1bf0-439c-9ba9-384318215caf","Type":"ContainerDied","Data":"000538556d7723e68ea587d9de6d291385d66869322ae11905b4802328d031bc"}
Sep 16 19:33:59.255231 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.255200    2578 scope.go:117] "RemoveContainer" containerID="1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"
Sep 16 19:33:59.263253 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.263237    2578 scope.go:117] "RemoveContainer" containerID="507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"
Sep 16 19:33:59.270862 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.270834    2578 scope.go:117] "RemoveContainer" containerID="e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"
Sep 16 19:33:59.278323 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.278305    2578 scope.go:117] "RemoveContainer" containerID="1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"
Sep 16 19:33:59.278590 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:33:59.278574    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33\": container with ID starting with 1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33 not found: ID does not exist" containerID="1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"
Sep 16 19:33:59.278634 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.278605    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33"} err="failed to get container status \"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33\": rpc error: code = NotFound desc = could not find container \"1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33\": container with ID starting with 1ae4b5bab450c846fd81b365b0e7f9d33cb8119e5ac2ed69eec443414e959e33 not found: ID does not exist"
Sep 16 19:33:59.278634 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.278621    2578 scope.go:117] "RemoveContainer" containerID="507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"
Sep 16 19:33:59.278897 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:33:59.278877    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc\": container with ID starting with 507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc not found: ID does not exist" containerID="507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"
Sep 16 19:33:59.278964 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.278904    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc"} err="failed to get container status \"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc\": rpc error: code = NotFound desc = could not find container \"507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc\": container with ID starting with 507b600701e827a3629cfca96f074f0e34e86121fe381f13f7eb45caeb1281dc not found: ID does not exist"
Sep 16 19:33:59.278964 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.278919    2578 scope.go:117] "RemoveContainer" containerID="e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"
Sep 16 19:33:59.279145 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:33:59.279130    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c\": container with ID starting with e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c not found: ID does not exist" containerID="e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"
Sep 16 19:33:59.279187 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.279149    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c"} err="failed to get container status \"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c\": rpc error: code = NotFound desc = could not find container \"e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c\": container with ID starting with e568af5c34df29a7482c1a02b262bb275426ca33c19062eacfaad9d220fded6c not found: ID does not exist"
Sep 16 19:33:59.282496 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.282473    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:33:59.286569 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.286544    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-578d87b44-t7ncr"]
Sep 16 19:33:59.324278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.324253    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5555c3f6-1bf0-439c-9ba9-384318215caf-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:33:59.324278 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.324274    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:33:59.324394 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.324286    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgz6\" (UniqueName: \"kubernetes.io/projected/5555c3f6-1bf0-439c-9ba9-384318215caf-kube-api-access-xtgz6\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:33:59.324394 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.324295    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5555c3f6-1bf0-439c-9ba9-384318215caf-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:33:59.324394 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.324303    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5555c3f6-1bf0-439c-9ba9-384318215caf-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:33:59.602627 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:33:59.602598    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" path="/var/lib/kubelet/pods/5555c3f6-1bf0-439c-9ba9-384318215caf/volumes"
Sep 16 19:34:02.269948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:02.269841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/0.log"
Sep 16 19:34:02.269948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:02.269909    2578 generic.go:356] "Generic (PLEG): container finished" podID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerID="a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171" exitCode=1
Sep 16 19:34:02.269948 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:02.269937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerDied","Data":"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"}
Sep 16 19:34:03.276920 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:03.276889    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/0.log"
Sep 16 19:34:03.277289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:03.276994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerStarted","Data":"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"}
Sep 16 19:34:05.367108 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:05.367070    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:34:05.367504 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:05.367307    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer" containerID="cri-o://ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5" gracePeriod=30
Sep 16 19:34:09.203787 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.203765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/1.log"
Sep 16 19:34:09.204230 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.204211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/0.log"
Sep 16 19:34:09.206572 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.206552    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:34:09.304827 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.304755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/1.log"
Sep 16 19:34:09.305226 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4_50d1a5f3-e425-481a-b007-0abb34b5829e/storage-initializer/0.log"
Sep 16 19:34:09.305286 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305240    2578 generic.go:356] "Generic (PLEG): container finished" podID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerID="ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5" exitCode=1
Sep 16 19:34:09.305324 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerDied","Data":"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"}
Sep 16 19:34:09.305324 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"
Sep 16 19:34:09.305398 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4" event={"ID":"50d1a5f3-e425-481a-b007-0abb34b5829e","Type":"ContainerDied","Data":"4068ca2ce09ac7bb86aeeabe4445486aec485a6bd383bbf6f7ba6208945496c3"}
Sep 16 19:34:09.305398 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.305363    2578 scope.go:117] "RemoveContainer" containerID="ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"
Sep 16 19:34:09.307498 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307478    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config\") pod \"50d1a5f3-e425-481a-b007-0abb34b5829e\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") "
Sep 16 19:34:09.307603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls\") pod \"50d1a5f3-e425-481a-b007-0abb34b5829e\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") "
Sep 16 19:34:09.307603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location\") pod \"50d1a5f3-e425-481a-b007-0abb34b5829e\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") "
Sep 16 19:34:09.307603 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw\") pod \"50d1a5f3-e425-481a-b007-0abb34b5829e\" (UID: \"50d1a5f3-e425-481a-b007-0abb34b5829e\") "
Sep 16 19:34:09.307819 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50d1a5f3-e425-481a-b007-0abb34b5829e" (UID: "50d1a5f3-e425-481a-b007-0abb34b5829e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:09.307954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50d1a5f3-e425-481a-b007-0abb34b5829e" (UID: "50d1a5f3-e425-481a-b007-0abb34b5829e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:09.307954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307917    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:34:09.307954 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.307930    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d1a5f3-e425-481a-b007-0abb34b5829e-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:34:09.309726 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.309700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50d1a5f3-e425-481a-b007-0abb34b5829e" (UID: "50d1a5f3-e425-481a-b007-0abb34b5829e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:09.309817 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.309792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw" (OuterVolumeSpecName: "kube-api-access-qxlnw") pod "50d1a5f3-e425-481a-b007-0abb34b5829e" (UID: "50d1a5f3-e425-481a-b007-0abb34b5829e"). InnerVolumeSpecName "kube-api-access-qxlnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:09.323339 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.323316    2578 scope.go:117] "RemoveContainer" containerID="a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"
Sep 16 19:34:09.331050 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.331027    2578 scope.go:117] "RemoveContainer" containerID="ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"
Sep 16 19:34:09.331366 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:34:09.331344    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5\": container with ID starting with ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5 not found: ID does not exist" containerID="ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"
Sep 16 19:34:09.331431 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.331374    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5"} err="failed to get container status \"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5\": rpc error: code = NotFound desc = could not find container \"ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5\": container with ID starting with ce35b0e567a363303b82d68973926cccdec47fce8276c8e6f7863d665d4441f5 not found: ID does not exist"
Sep 16 19:34:09.331431 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.331393    2578 scope.go:117] "RemoveContainer" containerID="a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"
Sep 16 19:34:09.331660 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:34:09.331643    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171\": container with ID starting with a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171 not found: ID does not exist" containerID="a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"
Sep 16 19:34:09.331705 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.331666    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171"} err="failed to get container status \"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171\": rpc error: code = NotFound desc = could not find container \"a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171\": container with ID starting with a32f19654e5105a40a5e18837b12220184b3aff6df4d40933489aa45a6ac2171 not found: ID does not exist"
Sep 16 19:34:09.408328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.408291    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/50d1a5f3-e425-481a-b007-0abb34b5829e-kube-api-access-qxlnw\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:34:09.408328 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.408325    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d1a5f3-e425-481a-b007-0abb34b5829e-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:34:09.641544 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.641509    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:34:09.646770 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:09.646735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6859c4cdb-rsbr4"]
Sep 16 19:34:11.604298 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:34:11.604264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" path="/var/lib/kubelet/pods/50d1a5f3-e425-481a-b007-0abb34b5829e/volumes"
Sep 16 19:35:22.263588 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263826    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kube-rbac-proxy"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kube-rbac-proxy"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263843    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263879    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263921    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263929    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kube-rbac-proxy"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263936    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5555c3f6-1bf0-439c-9ba9-384318215caf" containerName="kserve-container"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263989    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.264022 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.263994    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.264454 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.264035    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d1a5f3-e425-481a-b007-0abb34b5829e" containerName="storage-initializer"
Sep 16 19:35:22.267065 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.267049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.269694 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.269655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 16 19:35:22.269962 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.269942    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:22.270081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.269973    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:35:22.270081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.269975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:22.270081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.270021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:35:22.270081 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.270044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2m2z5\""
Sep 16 19:35:22.271158 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.271141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:35:22.276650 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.276630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:35:22.394927 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.394893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.395096 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.394935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.395096 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.394972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.395096 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.395022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.395096 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.395065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqmn\" (UniqueName: \"kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.495780 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.495751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.495975 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.495789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.495975 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.495825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.495975 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.495895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.495975 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.495927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqmn\" (UniqueName: \"kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.496390 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.496360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.496554 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.496534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.496791 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.496581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.498445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.498429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.505586 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.505562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqmn\" (UniqueName: \"kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.577734 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.577667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:22.850243 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:22.850215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:35:22.852925 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:35:22.852893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae8e40d_3f88_4114_8f6d_5b4b3038aeca.slice/crio-60cc36dcf807c325a2c3e74a18d987db93698d940abea49bc97d72b8a6e023c3 WatchSource:0}: Error finding container 60cc36dcf807c325a2c3e74a18d987db93698d940abea49bc97d72b8a6e023c3: Status 404 returned error can't find the container with id 60cc36dcf807c325a2c3e74a18d987db93698d940abea49bc97d72b8a6e023c3
Sep 16 19:35:23.607024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:23.606991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerStarted","Data":"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"}
Sep 16 19:35:23.607413 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:23.607044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerStarted","Data":"60cc36dcf807c325a2c3e74a18d987db93698d940abea49bc97d72b8a6e023c3"}
Sep 16 19:35:27.625292 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:27.625258    2578 generic.go:356] "Generic (PLEG): container finished" podID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerID="64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5" exitCode=0
Sep 16 19:35:27.625682 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:27.625330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerDied","Data":"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"}
Sep 16 19:35:27.626468 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:27.626452    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:35:35.665712 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:35.665679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerStarted","Data":"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"}
Sep 16 19:35:35.666116 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:35.665735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerStarted","Data":"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"}
Sep 16 19:35:35.666116 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:35.665913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:35.685073 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:35.685022    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podStartSLOduration=2.4776947209999998 podStartE2EDuration="13.685008366s" totalImagesPullingTime="7.159102181s" totalInitContainerRuntime="4.048211464s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:35.684830711 +0000 UTC m=+2994.656307962" watchObservedRunningTime="2026-09-16 19:35:35.685008366 +0000 UTC m=+2994.656485658"
Sep 16 19:35:36.669997 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:36.669964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:36.671040 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:36.671019    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:35:37.673688 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:37.673653    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:35:42.678942 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:42.678911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:35:42.679320 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:42.679304    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:35:52.679445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:35:52.679404    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:02.679541 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:02.679501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:12.680031 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:12.679996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:22.679451 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:22.679407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:32.679555 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:32.679467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:42.679411 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:42.679371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:36:52.599017 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:36:52.598988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:37:03.271959 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:03.271910    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:37:03.272498 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:03.272231    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container" containerID="cri-o://1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd" gracePeriod=30
Sep 16 19:37:03.272498 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:03.272388    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kube-rbac-proxy" containerID="cri-o://d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2" gracePeriod=30
Sep 16 19:37:04.031011 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:04.030978    2578 generic.go:356] "Generic (PLEG): container finished" podID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerID="d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2" exitCode=2
Sep 16 19:37:04.031185 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:04.031021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerDied","Data":"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"}
Sep 16 19:37:06.922005 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.921978    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:37:06.963218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert\") pod \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") "
Sep 16 19:37:06.963218 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls\") pod \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") "
Sep 16 19:37:06.963392 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location\") pod \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") "
Sep 16 19:37:06.963392 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpqmn\" (UniqueName: \"kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn\") pod \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") "
Sep 16 19:37:06.963392 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config\") pod \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\" (UID: \"eae8e40d-3f88-4114-8f6d-5b4b3038aeca\") "
Sep 16 19:37:06.963609 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eae8e40d-3f88-4114-8f6d-5b4b3038aeca" (UID: "eae8e40d-3f88-4114-8f6d-5b4b3038aeca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:06.963692 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eae8e40d-3f88-4114-8f6d-5b4b3038aeca" (UID: "eae8e40d-3f88-4114-8f6d-5b4b3038aeca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:06.963790 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.963769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eae8e40d-3f88-4114-8f6d-5b4b3038aeca" (UID: "eae8e40d-3f88-4114-8f6d-5b4b3038aeca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:06.965456 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.965430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eae8e40d-3f88-4114-8f6d-5b4b3038aeca" (UID: "eae8e40d-3f88-4114-8f6d-5b4b3038aeca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:06.965573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:06.965545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn" (OuterVolumeSpecName: "kube-api-access-cpqmn") pod "eae8e40d-3f88-4114-8f6d-5b4b3038aeca" (UID: "eae8e40d-3f88-4114-8f6d-5b4b3038aeca"). InnerVolumeSpecName "kube-api-access-cpqmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:07.046018 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.045939    2578 generic.go:356] "Generic (PLEG): container finished" podID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerID="1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd" exitCode=0
Sep 16 19:37:07.046165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.046038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerDied","Data":"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"}
Sep 16 19:37:07.046165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.046062    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"
Sep 16 19:37:07.046165 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.046139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc" event={"ID":"eae8e40d-3f88-4114-8f6d-5b4b3038aeca","Type":"ContainerDied","Data":"60cc36dcf807c325a2c3e74a18d987db93698d940abea49bc97d72b8a6e023c3"}
Sep 16 19:37:07.046289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.046179    2578 scope.go:117] "RemoveContainer" containerID="d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"
Sep 16 19:37:07.054925 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.054905    2578 scope.go:117] "RemoveContainer" containerID="1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"
Sep 16 19:37:07.062303 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.062286    2578 scope.go:117] "RemoveContainer" containerID="64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"
Sep 16 19:37:07.064443 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.064425    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:07.064506 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.064446    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-cabundle-cert\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:07.064506 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.064456    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-proxy-tls\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:07.064506 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.064464    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kserve-provision-location\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:07.064506 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.064472    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpqmn\" (UniqueName: \"kubernetes.io/projected/eae8e40d-3f88-4114-8f6d-5b4b3038aeca-kube-api-access-cpqmn\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:07.068321 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.068300    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:37:07.070735 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.070629    2578 scope.go:117] "RemoveContainer" containerID="d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"
Sep 16 19:37:07.070965 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:07.070943    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2\": container with ID starting with d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2 not found: ID does not exist" containerID="d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"
Sep 16 19:37:07.071041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.070976    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2"} err="failed to get container status \"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2\": rpc error: code = NotFound desc = could not find container \"d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2\": container with ID starting with d6657a4221ad907bbeac9b7fdbf03968f9fd978e6d1c2a2e25f76889929e44a2 not found: ID does not exist"
Sep 16 19:37:07.071041 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.070999    2578 scope.go:117] "RemoveContainer" containerID="1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"
Sep 16 19:37:07.071271 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:07.071244    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd\": container with ID starting with 1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd not found: ID does not exist" containerID="1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"
Sep 16 19:37:07.071316 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.071277    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd"} err="failed to get container status \"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd\": rpc error: code = NotFound desc = could not find container \"1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd\": container with ID starting with 1c3a4f8c5fbfe996113690d985043ebae721a838f7d7c14864538ad2be9540cd not found: ID does not exist"
Sep 16 19:37:07.071316 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.071298    2578 scope.go:117] "RemoveContainer" containerID="64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"
Sep 16 19:37:07.071601 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:07.071582    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5\": container with ID starting with 64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5 not found: ID does not exist" containerID="64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"
Sep 16 19:37:07.071686 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.071660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5"} err="failed to get container status \"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5\": rpc error: code = NotFound desc = could not find container \"64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5\": container with ID starting with 64761db929b282c0fe96ec9bf575fa211e72cec9eab05e1e5b0124e12a9869c5 not found: ID does not exist"
Sep 16 19:37:07.072351 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.072333    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pqzsc"]
Sep 16 19:37:07.602807 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:07.602772    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" path="/var/lib/kubelet/pods/eae8e40d-3f88-4114-8f6d-5b4b3038aeca/volumes"
Sep 16 19:37:10.236909 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.236871    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fkmrx/must-gather-q7vx6"]
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kube-rbac-proxy"
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237314    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kube-rbac-proxy"
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="storage-initializer"
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="storage-initializer"
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container"
Sep 16 19:37:10.237384 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container"
Sep 16 19:37:10.237702 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237446    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kube-rbac-proxy"
Sep 16 19:37:10.237702 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.237462    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae8e40d-3f88-4114-8f6d-5b4b3038aeca" containerName="kserve-container"
Sep 16 19:37:10.242594 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.242570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.245573 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.245553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkmrx\"/\"kube-root-ca.crt\""
Sep 16 19:37:10.246767 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.246745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkmrx\"/\"openshift-service-ca.crt\""
Sep 16 19:37:10.246940 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.246746    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fkmrx\"/\"default-dockercfg-lf29s\""
Sep 16 19:37:10.248556 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.248537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkmrx/must-gather-q7vx6"]
Sep 16 19:37:10.292743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.292707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.292743 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.292742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpf7k\" (UniqueName: \"kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.393388 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.393352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.393388 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.393388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpf7k\" (UniqueName: \"kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.393690 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.393671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.402419 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.402389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpf7k\" (UniqueName: \"kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k\") pod \"must-gather-q7vx6\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") " pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.560846 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.560749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:10.830289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:10.830266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkmrx/must-gather-q7vx6"]
Sep 16 19:37:10.832676 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:37:10.832645    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43f7e575_de51_46fb_9fdb_4870c9c20f82.slice/crio-dbc1d60dafb53b707460dcdab4d6e310fe5452aedbfcd46f6b789395200c116a WatchSource:0}: Error finding container dbc1d60dafb53b707460dcdab4d6e310fe5452aedbfcd46f6b789395200c116a: Status 404 returned error can't find the container with id dbc1d60dafb53b707460dcdab4d6e310fe5452aedbfcd46f6b789395200c116a
Sep 16 19:37:11.067924 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:11.067889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" event={"ID":"43f7e575-de51-46fb-9fdb-4870c9c20f82","Type":"ContainerStarted","Data":"dbc1d60dafb53b707460dcdab4d6e310fe5452aedbfcd46f6b789395200c116a"}
Sep 16 19:37:17.103448 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:17.103411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" event={"ID":"43f7e575-de51-46fb-9fdb-4870c9c20f82","Type":"ContainerStarted","Data":"9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312"}
Sep 16 19:37:17.103874 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:17.103457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" event={"ID":"43f7e575-de51-46fb-9fdb-4870c9c20f82","Type":"ContainerStarted","Data":"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"}
Sep 16 19:37:17.129299 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:17.129249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" podStartSLOduration=1.200663198 podStartE2EDuration="7.129233623s" totalImagesPullingTime="5.928570425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:17.127628198 +0000 UTC m=+3096.099105463" watchObservedRunningTime="2026-09-16 19:37:17.129233623 +0000 UTC m=+3096.100710937"
Sep 16 19:37:48.244599 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:48.244564    2578 generic.go:356] "Generic (PLEG): container finished" podID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerID="bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2" exitCode=0
Sep 16 19:37:48.245056 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:48.244639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" event={"ID":"43f7e575-de51-46fb-9fdb-4870c9c20f82","Type":"ContainerDied","Data":"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"}
Sep 16 19:37:48.245056 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:48.244979    2578 scope.go:117] "RemoveContainer" containerID="bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"
Sep 16 19:37:48.615316 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:48.615270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fkmrx_must-gather-q7vx6_43f7e575-de51-46fb-9fdb-4870c9c20f82/gather/0.log"
Sep 16 19:37:49.530289 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.530258    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wwtss/must-gather-rs29g"]
Sep 16 19:37:49.532866 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.532829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.537906 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.537881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwtss\"/\"kube-root-ca.crt\""
Sep 16 19:37:49.538273 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.537883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwtss\"/\"openshift-service-ca.crt\""
Sep 16 19:37:49.538273 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.537886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wwtss\"/\"default-dockercfg-227rs\""
Sep 16 19:37:49.542971 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.542949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/must-gather-rs29g"]
Sep 16 19:37:49.640628 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.640584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f57d4b1-3abb-46c0-90db-97364971765b-must-gather-output\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.640830 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.640643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4hx\" (UniqueName: \"kubernetes.io/projected/9f57d4b1-3abb-46c0-90db-97364971765b-kube-api-access-vz4hx\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.741531 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.741494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f57d4b1-3abb-46c0-90db-97364971765b-must-gather-output\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.741531 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.741532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4hx\" (UniqueName: \"kubernetes.io/projected/9f57d4b1-3abb-46c0-90db-97364971765b-kube-api-access-vz4hx\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.741912 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.741891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f57d4b1-3abb-46c0-90db-97364971765b-must-gather-output\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.750625 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.750594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4hx\" (UniqueName: \"kubernetes.io/projected/9f57d4b1-3abb-46c0-90db-97364971765b-kube-api-access-vz4hx\") pod \"must-gather-rs29g\" (UID: \"9f57d4b1-3abb-46c0-90db-97364971765b\") " pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:49.842643 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:49.842611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/must-gather-rs29g"
Sep 16 19:37:50.112745 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:50.112720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/must-gather-rs29g"]
Sep 16 19:37:50.114605 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:37:50.114557    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f57d4b1_3abb_46c0_90db_97364971765b.slice/crio-17e7106f7e2db6b51c01e4ce850a119a9ac4f29a19a68e61a3effdcfd03f6f41 WatchSource:0}: Error finding container 17e7106f7e2db6b51c01e4ce850a119a9ac4f29a19a68e61a3effdcfd03f6f41: Status 404 returned error can't find the container with id 17e7106f7e2db6b51c01e4ce850a119a9ac4f29a19a68e61a3effdcfd03f6f41
Sep 16 19:37:50.253342 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:50.253309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/must-gather-rs29g" event={"ID":"9f57d4b1-3abb-46c0-90db-97364971765b","Type":"ContainerStarted","Data":"17e7106f7e2db6b51c01e4ce850a119a9ac4f29a19a68e61a3effdcfd03f6f41"}
Sep 16 19:37:51.262243 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:51.262196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/must-gather-rs29g" event={"ID":"9f57d4b1-3abb-46c0-90db-97364971765b","Type":"ContainerStarted","Data":"e768f018e35f6449e6dd3fcbaecb290004bb936695387de3b36f3d48b3ab2c77"}
Sep 16 19:37:51.262243 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:51.262244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/must-gather-rs29g" event={"ID":"9f57d4b1-3abb-46c0-90db-97364971765b","Type":"ContainerStarted","Data":"09b7e01e4f9335aee68cb663511cfc8a89c9c2853bee6270f91eee5e8903ed57"}
Sep 16 19:37:51.279349 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:51.279285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wwtss/must-gather-rs29g" podStartSLOduration=1.6407386640000001 podStartE2EDuration="2.279264947s" totalImagesPullingTime="638.526283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:51.278145791 +0000 UTC m=+3130.249623043" watchObservedRunningTime="2026-09-16 19:37:51.279264947 +0000 UTC m=+3130.250742198"
Sep 16 19:37:52.295257 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:52.295224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mwm6b_96514ca7-0140-4654-9b6c-7c16f636afd6/global-pull-secret-syncer/0.log"
Sep 16 19:37:52.382058 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:52.382023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-crttb_9f35ea27-56a7-4718-9e06-4fbfdb490581/konnectivity-agent/0.log"
Sep 16 19:37:52.465062 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:52.465024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-130.ec2.internal_d7e51c5c4cf47eed84f85b77e7cf2641/haproxy/0.log"
Sep 16 19:37:54.355582 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.355546    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-fkmrx/must-gather-q7vx6"]
Sep 16 19:37:54.356008 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.355768    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-fkmrx/must-gather-q7vx6" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="copy" containerID="cri-o://9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312" gracePeriod=2
Sep 16 19:37:54.358662 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:54.358610    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-q7vx6\" is forbidden: User \"system:node:ip-10-0-142-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fkmrx\": no relationship found between node 'ip-10-0-142-130.ec2.internal' and this object" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:54.360193 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.359792    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-fkmrx/must-gather-q7vx6"]
Sep 16 19:37:54.583487 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.583453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fkmrx_must-gather-q7vx6_43f7e575-de51-46fb-9fdb-4870c9c20f82/copy/0.log"
Sep 16 19:37:54.586210 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.586190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:54.589095 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:54.589065    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-q7vx6\" is forbidden: User \"system:node:ip-10-0-142-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fkmrx\": no relationship found between node 'ip-10-0-142-130.ec2.internal' and this object" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:54.684522 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.684436    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpf7k\" (UniqueName: \"kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k\") pod \"43f7e575-de51-46fb-9fdb-4870c9c20f82\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") "
Sep 16 19:37:54.684673 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.684525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output\") pod \"43f7e575-de51-46fb-9fdb-4870c9c20f82\" (UID: \"43f7e575-de51-46fb-9fdb-4870c9c20f82\") "
Sep 16 19:37:54.685701 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.685670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "43f7e575-de51-46fb-9fdb-4870c9c20f82" (UID: "43f7e575-de51-46fb-9fdb-4870c9c20f82"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:54.686842 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.686821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k" (OuterVolumeSpecName: "kube-api-access-tpf7k") pod "43f7e575-de51-46fb-9fdb-4870c9c20f82" (UID: "43f7e575-de51-46fb-9fdb-4870c9c20f82"). InnerVolumeSpecName "kube-api-access-tpf7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:54.785802 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.785765    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43f7e575-de51-46fb-9fdb-4870c9c20f82-must-gather-output\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:54.785802 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:54.785796    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpf7k\" (UniqueName: \"kubernetes.io/projected/43f7e575-de51-46fb-9fdb-4870c9c20f82-kube-api-access-tpf7k\") on node \"ip-10-0-142-130.ec2.internal\" DevicePath \"\""
Sep 16 19:37:55.280434 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.280395    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fkmrx_must-gather-q7vx6_43f7e575-de51-46fb-9fdb-4870c9c20f82/copy/0.log"
Sep 16 19:37:55.280903 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.280875    2578 generic.go:356] "Generic (PLEG): container finished" podID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerID="9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312" exitCode=143
Sep 16 19:37:55.281064 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.280918    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:55.281064 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.280958    2578 scope.go:117] "RemoveContainer" containerID="9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312"
Sep 16 19:37:55.283557 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:55.283518    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-q7vx6\" is forbidden: User \"system:node:ip-10-0-142-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fkmrx\": no relationship found between node 'ip-10-0-142-130.ec2.internal' and this object" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:55.290227 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.290201    2578 scope.go:117] "RemoveContainer" containerID="bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"
Sep 16 19:37:55.292093 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:55.292059    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-q7vx6\" is forbidden: User \"system:node:ip-10-0-142-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fkmrx\": no relationship found between node 'ip-10-0-142-130.ec2.internal' and this object" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" pod="openshift-must-gather-fkmrx/must-gather-q7vx6"
Sep 16 19:37:55.302072 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.302053    2578 scope.go:117] "RemoveContainer" containerID="9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312"
Sep 16 19:37:55.302388 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:55.302360    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312\": container with ID starting with 9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312 not found: ID does not exist" containerID="9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312"
Sep 16 19:37:55.302495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.302399    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312"} err="failed to get container status \"9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312\": rpc error: code = NotFound desc = could not find container \"9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312\": container with ID starting with 9a020b511cb8e2e64b61f86f2be695310f061bc45de9938a7434f8ba567bb312 not found: ID does not exist"
Sep 16 19:37:55.302495 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.302421    2578 scope.go:117] "RemoveContainer" containerID="bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"
Sep 16 19:37:55.302776 ip-10-0-142-130 kubenswrapper[2578]: E0916 19:37:55.302734    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2\": container with ID starting with bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2 not found: ID does not exist" containerID="bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"
Sep 16 19:37:55.302879 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.302768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2"} err="failed to get container status \"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2\": rpc error: code = NotFound desc = could not find container \"bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2\": container with ID starting with bd961368b37c3923b0ffaa146cb5bf77b1a1320d8e17e22bff35c90514bf1ff2 not found: ID does not exist"
Sep 16 19:37:55.607053 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:55.607010    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" path="/var/lib/kubelet/pods/43f7e575-de51-46fb-9fdb-4870c9c20f82/volumes"
Sep 16 19:37:58.742246 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.742215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z57gz_915c4408-d3bd-431c-8d23-f522f45d8f46/kube-state-metrics/0.log"
Sep 16 19:37:58.775435 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.775371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z57gz_915c4408-d3bd-431c-8d23-f522f45d8f46/kube-rbac-proxy-main/0.log"
Sep 16 19:37:58.802936 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.802901    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-z57gz_915c4408-d3bd-431c-8d23-f522f45d8f46/kube-rbac-proxy-self/0.log"
Sep 16 19:37:58.902379 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.902340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwrg6_0856dc1a-4dba-46a3-9cfb-858b3e515f5e/node-exporter/0.log"
Sep 16 19:37:58.930714 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.930136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwrg6_0856dc1a-4dba-46a3-9cfb-858b3e515f5e/kube-rbac-proxy/0.log"
Sep 16 19:37:58.973319 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:58.973281    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwrg6_0856dc1a-4dba-46a3-9cfb-858b3e515f5e/init-textfile/0.log"
Sep 16 19:37:59.331279 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.331241    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/prometheus/0.log"
Sep 16 19:37:59.370986 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.370959    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/config-reloader/0.log"
Sep 16 19:37:59.407549 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.407524    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/thanos-sidecar/0.log"
Sep 16 19:37:59.447145 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.447114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/kube-rbac-proxy-web/0.log"
Sep 16 19:37:59.488520 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.488490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/kube-rbac-proxy/0.log"
Sep 16 19:37:59.529332 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.529302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/kube-rbac-proxy-thanos/0.log"
Sep 16 19:37:59.553385 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.553361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d698d7e-0ecc-48b3-963b-08bb230d3124/init-config-reloader/0.log"
Sep 16 19:37:59.584179 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.584085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8tpfz_dffafd27-7d51-4189-83eb-c08dd7fcbb6a/prometheus-operator/0.log"
Sep 16 19:37:59.604203 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.604178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8tpfz_dffafd27-7d51-4189-83eb-c08dd7fcbb6a/kube-rbac-proxy/0.log"
Sep 16 19:37:59.639981 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.639957    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-vs7xn_8a23632f-7447-409a-968f-221a047ddb56/prometheus-operator-admission-webhook/0.log"
Sep 16 19:37:59.750826 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.750791    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/thanos-query/0.log"
Sep 16 19:37:59.774914 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.774887    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/kube-rbac-proxy-web/0.log"
Sep 16 19:37:59.801150 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.801126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/kube-rbac-proxy/0.log"
Sep 16 19:37:59.824222 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.824195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/prom-label-proxy/0.log"
Sep 16 19:37:59.847684 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.847655    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/kube-rbac-proxy-rules/0.log"
Sep 16 19:37:59.874445 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:37:59.874420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69454d8c7b-wzlzp_6a6569c2-e6d2-4456-a427-0926f57f11f3/kube-rbac-proxy-metrics/0.log"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.596922    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"]
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597366    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="gather"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597383    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="gather"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597397    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="copy"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="copy"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597508    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="gather"
Sep 16 19:38:02.598247 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.597518    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f7e575-de51-46fb-9fdb-4870c9c20f82" containerName="copy"
Sep 16 19:38:02.603824 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.603800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.610981 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.610949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"]
Sep 16 19:38:02.655222 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.655186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-podres\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.655502 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.655310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8r8r\" (UniqueName: \"kubernetes.io/projected/e1825db2-2d19-4336-8cda-311756a96bdb-kube-api-access-w8r8r\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.655502 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.655345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-proc\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.655502 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.655389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-lib-modules\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.655502 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.655435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-sys\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756338 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-sys\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-podres\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8r8r\" (UniqueName: \"kubernetes.io/projected/e1825db2-2d19-4336-8cda-311756a96bdb-kube-api-access-w8r8r\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-proc\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-lib-modules\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-sys\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-podres\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756519 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-proc\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.756886 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.756539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1825db2-2d19-4336-8cda-311756a96bdb-lib-modules\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.764945 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.764919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8r8r\" (UniqueName: \"kubernetes.io/projected/e1825db2-2d19-4336-8cda-311756a96bdb-kube-api-access-w8r8r\") pod \"perf-node-gather-daemonset-w7tvl\" (UID: \"e1825db2-2d19-4336-8cda-311756a96bdb\") " pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:02.918543 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:02.918449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:03.205910 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:03.205880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"]
Sep 16 19:38:03.209505 ip-10-0-142-130 kubenswrapper[2578]: W0916 19:38:03.208823    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1825db2_2d19_4336_8cda_311756a96bdb.slice/crio-6f05de1ddd0f3500840dbd0b3ca24515762cbd987d4c286198b4f9e54ecd9be6 WatchSource:0}: Error finding container 6f05de1ddd0f3500840dbd0b3ca24515762cbd987d4c286198b4f9e54ecd9be6: Status 404 returned error can't find the container with id 6f05de1ddd0f3500840dbd0b3ca24515762cbd987d4c286198b4f9e54ecd9be6
Sep 16 19:38:03.352163 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:03.352113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl" event={"ID":"e1825db2-2d19-4336-8cda-311756a96bdb","Type":"ContainerStarted","Data":"6f05de1ddd0f3500840dbd0b3ca24515762cbd987d4c286198b4f9e54ecd9be6"}
Sep 16 19:38:03.681942 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:03.681913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2wq27_71efd6a1-843c-4d6c-a064-2e62158db0ef/dns/0.log"
Sep 16 19:38:03.704213 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:03.704178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2wq27_71efd6a1-843c-4d6c-a064-2e62158db0ef/kube-rbac-proxy/0.log"
Sep 16 19:38:03.853676 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:03.853651    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d979r_cf5e4bc2-3bae-4b20-9f56-2ddc8acbe61b/dns-node-resolver/0.log"
Sep 16 19:38:04.342776 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:04.342729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-52748_9ef0ebe2-8dd0-4bbf-b7a2-3a00a7712069/node-ca/0.log"
Sep 16 19:38:04.358537 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:04.358496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl" event={"ID":"e1825db2-2d19-4336-8cda-311756a96bdb","Type":"ContainerStarted","Data":"1bc7b2800fe34ea686e3fba80a6995abf3cd271bf9250768fe16e838f4715d32"}
Sep 16 19:38:04.359382 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:04.359350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:04.379485 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:04.379432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl" podStartSLOduration=2.379415849 podStartE2EDuration="2.379415849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:04.376933451 +0000 UTC m=+3143.348410691" watchObservedRunningTime="2026-09-16 19:38:04.379415849 +0000 UTC m=+3143.350893097"
Sep 16 19:38:05.226179 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:05.226152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59898dc88-7qb62_ff078225-e337-491b-8f17-6d7464d9b1e4/router/0.log"
Sep 16 19:38:05.579075 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:05.578998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2pnvn_c4befc80-f5e8-4515-b8fb-94796b62d41e/serve-healthcheck-canary/0.log"
Sep 16 19:38:05.996575 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:05.996542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vzbs8_37201f6a-b634-4878-80a4-f04088a883db/insights-operator/0.log"
Sep 16 19:38:05.997451 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:05.997431    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vzbs8_37201f6a-b634-4878-80a4-f04088a883db/insights-operator/1.log"
Sep 16 19:38:06.092206 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:06.092173    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfpwz_9980b4f7-4c0e-4a6d-965d-3a92021e24e1/kube-rbac-proxy/0.log"
Sep 16 19:38:06.117181 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:06.117152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfpwz_9980b4f7-4c0e-4a6d-965d-3a92021e24e1/exporter/0.log"
Sep 16 19:38:06.139281 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:06.139251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfpwz_9980b4f7-4c0e-4a6d-965d-3a92021e24e1/extractor/0.log"
Sep 16 19:38:09.023987 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.023957    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qspfl_32f52297-3b6d-47a4-9bea-af6f98fbec8e/s3-init/0.log"
Sep 16 19:38:09.041294 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.041259    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qspfl_32f52297-3b6d-47a4-9bea-af6f98fbec8e/download-hf-model/0.log"
Sep 16 19:38:09.070129 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.070104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2mxn8_92f5f471-dbb9-4eb7-a347-57364e1cf06a/s3-tls-init-custom/0.log"
Sep 16 19:38:09.086738 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.086717    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2mxn8_92f5f471-dbb9-4eb7-a347-57364e1cf06a/download-hf-model/0.log"
Sep 16 19:38:09.116661 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.116639    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-r9d4r_c9c916d7-6c42-4a06-b44a-a3591e976453/s3-tls-init-serving/0.log"
Sep 16 19:38:09.134656 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:09.134636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-r9d4r_c9c916d7-6c42-4a06-b44a-a3591e976453/download-hf-model/0.log"
Sep 16 19:38:11.380107 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:11.379711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wwtss/perf-node-gather-daemonset-w7tvl"
Sep 16 19:38:13.636417 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:13.636391    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wr57z_722e7455-366c-4769-be97-2bbed814df49/migrator/0.log"
Sep 16 19:38:13.661279 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:13.661240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wr57z_722e7455-366c-4769-be97-2bbed814df49/graceful-termination/0.log"
Sep 16 19:38:14.969883 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:14.969546    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-55hqc_e2554090-4b49-4158-9277-23fd3dbe8ac7/kube-multus/0.log"
Sep 16 19:38:15.222666 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.222592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:38:15.245482 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.245453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/egress-router-binary-copy/0.log"
Sep 16 19:38:15.270480 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.270455    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/cni-plugins/0.log"
Sep 16 19:38:15.295399 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.295370    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/bond-cni-plugin/0.log"
Sep 16 19:38:15.317271 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.317225    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/routeoverride-cni/0.log"
Sep 16 19:38:15.340889 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.340867    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/whereabouts-cni-bincopy/0.log"
Sep 16 19:38:15.367698 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.367669    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wc6zk_2b7e81bb-1f13-4f36-a344-952e6946dd68/whereabouts-cni/0.log"
Sep 16 19:38:15.658999 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.658969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zpmsl_e86c725d-a1ce-429e-8073-83919fcff64b/network-metrics-daemon/0.log"
Sep 16 19:38:15.683412 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:15.683378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zpmsl_e86c725d-a1ce-429e-8073-83919fcff64b/kube-rbac-proxy/0.log"
Sep 16 19:38:16.471327 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.471295    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/ovn-controller/0.log"
Sep 16 19:38:16.488838 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.488814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/ovn-acl-logging/0.log"
Sep 16 19:38:16.513427 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.513402    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/kube-rbac-proxy-node/0.log"
Sep 16 19:38:16.538319 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.538296    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:38:16.563523 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.563503    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/northd/0.log"
Sep 16 19:38:16.585283 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.585258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/nbdb/0.log"
Sep 16 19:38:16.608024 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.607997    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/sbdb/0.log"
Sep 16 19:38:16.664798 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:16.664761    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5xh5m_a1a862c7-4d03-440d-a483-b0dfb7f21042/ovnkube-controller/0.log"
Sep 16 19:38:18.198778 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:18.198747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xsm6d_3f558960-469f-40f1-9b7d-2a9208bc2260/network-check-target-container/0.log"
Sep 16 19:38:19.164095 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:19.164065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vpqf8_6750ad10-4915-4413-a651-f22681422251/iptables-alerter/0.log"
Sep 16 19:38:19.908228 ip-10-0-142-130 kubenswrapper[2578]: I0916 19:38:19.908198    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lvsmf_c614891f-8a28-49d4-994f-30fc9308cfaf/tuned/0.log"