Sep 16 18:41:40.346213 ip-10-0-130-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:41:40.346224 ip-10-0-130-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:41:40.346231 ip-10-0-130-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:41:40.346377 ip-10-0-130-142 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:41:50.501849 ip-10-0-130-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:41:50.501867 ip-10-0-130-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e23edaa574844bf59c9ae0e154e749d5 --
Sep 16 18:43:43.738666 ip-10-0-130-142 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:44.102985 ip-10-0-130-142 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:44.102985 ip-10-0-130-142 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:44.102985 ip-10-0-130-142 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:44.102985 ip-10-0-130-142 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:44.102985 ip-10-0-130-142 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:44.103913 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.103829    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:44.109369 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109354    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.109369 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109367    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.109369 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109371    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109393    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109399    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109402    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109405    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109408    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109412    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109415    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109418    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109421    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109424    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109427    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109430    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109433    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109436    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109439    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109441    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109444    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109448    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.109468 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109452    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109456    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109458    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109462    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109465    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109467    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109470    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109473    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109476    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109479    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109482    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109485    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109488    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109492    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109495    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109498    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109501    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109504    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109506    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109509    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.109921 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109512    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109515    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109518    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109520    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109523    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109526    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109528    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109531    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109533    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109536    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109538    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109541    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109543    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109546    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109549    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109552    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109555    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109558    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109561    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.110427 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109564    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109567    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109569    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109572    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109574    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109577    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109580    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109583    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109586    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109589    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109592    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109594    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109597    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109599    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109602    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109604    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109607    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109610    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109612    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109615    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.110925 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109618    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109622    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109625    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109627    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109630    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109632    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109635    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109639    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109642    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109645    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109649    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109653    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109656    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109659    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109661    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109664    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109666    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109669    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.109672    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110339    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.111725 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110345    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110348    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110351    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110354    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110357    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110359    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110362    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110365    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110367    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110370    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110373    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110375    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110378    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110380    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110396    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110399    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110402    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110405    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110407    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110413    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.112223 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110417    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110421    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110424    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110428    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110431    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110434    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110436    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110439    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110442    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110445    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110447    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110450    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110452    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110455    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110457    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110461    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110463    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110466    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110469    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.112837 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110471    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110474    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110477    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110479    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110482    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110484    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110487    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110489    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110493    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110496    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110498    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110501    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110505    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110508    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110511    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110514    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110517    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110521    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110524    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110527    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.113313 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110529    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110532    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110535    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110538    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110541    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110543    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110546    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110548    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110550    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110554    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110557    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110560    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110562    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110565    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110567    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110570    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110572    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110575    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110578    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110580    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.113830 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110583    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110586    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110589    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110591    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110594    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110597    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110600    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110602    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110605    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110608    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110611    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110615    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110618    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110621    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110624    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110626    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110629    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110632    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.110634    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.114323 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113018    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113032    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113036    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113041    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113046    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113049    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113052    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113056    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113059    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113062    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113064    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113067    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113070    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113072    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113075    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113078    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113081    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113085    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113088    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.114804 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113090    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113094    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113096    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113099    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113102    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113104    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113107    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113109    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113112    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113115    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113117    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113120    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113123    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113126    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113129    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113132    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113143    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113146    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113149    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113152    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.115263 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113154    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113157    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113160    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113162    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113165    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113167    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113171    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113174    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113176    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113179    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113182    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113187    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113189    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113192    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113195    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113197    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113200    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113203    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113205    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.115760 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113208    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113211    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113213    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113216    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113219    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113223    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113227    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113231    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113233    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113236    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113239    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113241    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113244    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113247    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113250    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113252    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113255    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113257    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113260    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113263    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.116252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113265    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113268    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113271    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113274    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113277    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113280    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113283    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113286    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113290    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113293    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113296    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113299    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113302    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113305    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113307    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113310    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113313    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113316    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113319    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.116747 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113322    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.113325    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113411    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113455    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113462    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113467    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113471    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113474    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113479    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113483    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113487    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113490    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113493    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113496    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113500    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113503    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113506    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113509    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113512    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113516    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113520    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113522    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113525    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113528    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:44.117208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113532    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113535    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113538    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113542    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113545    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113548    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113551    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113554    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113557    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113560    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113564    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113567    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113570    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113573    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113576    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113579    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113583    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113587    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113589    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113592    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113595    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113599    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113602    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113605    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113608    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:44.117788 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113611    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113614    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113617    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113620    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113623    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113626    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113629    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113632    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113636    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113639    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113642    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113645    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113648    2581 flags.go:64] FLAG: --help="false"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113651    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113654    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113657    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113660    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113664    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113667    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113670    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113673    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113676    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113679    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:44.118391 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113682    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113685    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113687    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113690    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113693    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113696    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113699    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113702    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113705    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113708    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113712    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113717    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113720    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113723    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113726    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113729    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113733    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113736    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113738    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113743    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113746    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113751    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113754    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113757    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113760    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:44.118989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113763    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113766    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113769    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113772    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113780    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113783    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113786    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113789    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113792    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113797    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113800    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113803    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113806    2581 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113809    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113812    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a25650321b1334b3"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113815    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113818    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113821    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113824    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113828    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113831    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113834    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113837    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113841    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113844    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:44.119643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113847    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113850    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113853    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113856    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113859    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113866    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113869    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113872    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113875    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113878    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113881    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113886    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113889    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113892    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113895    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113898    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113901    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113904    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113910    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113913    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113916    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113921    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113924    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113927    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113929    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:44.120264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113932    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113935    2581 flags.go:64] FLAG: --v="2"
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113939    2581 flags.go:64] FLAG: --version="false"
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113943    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113948    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.113951    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114058    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114062    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114065    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114068    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114071    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114074    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114078    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114081    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114084    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114088    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114091    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114094    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114098    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114102    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114105    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.120914 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114108    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114111    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114114    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114117    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114120    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114123    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114126    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114129    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114131    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114134    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114137    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114140    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114142    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114145    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114147    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114150    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114154    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114157    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114159    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.121545 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114162    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114164    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114167    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114171    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114176    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114179    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114182    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114185    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114189    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114192    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114196    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114199    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114202    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114205    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114207    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114210    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114212    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114215    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114218    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.122018 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114220    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114223    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114226    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114228    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114232    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114236    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114239    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114242    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114244    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114248    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114250    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114254    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114257    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114260    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114262    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114265    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114268    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114272    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.122499 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114279    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114282    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114285    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114288    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114291    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114294    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114297    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114300    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114303    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114305    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114308    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114311    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114314    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114317    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114319    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114322    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114325    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114327    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114330    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114333    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.122956 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114336    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114339    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114341    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114344    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114347    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114350    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.114353    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.114868    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.121940    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.121954    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.122616    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.122623    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122680    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122685    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122688    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.123516 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122691    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122694    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122697    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122701    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122704    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122706    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122709    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122712    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122714    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122717    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122720    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122723    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122725    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122728    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122731    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122734    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122736    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122739    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122742    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122744    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.123898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122747    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122749    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122752    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122755    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122757    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122760    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122764    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122767    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122770    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122773    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122776    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122778    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122781    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122784    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122787    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122789    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122792    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122796    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122800    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122803    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.124406 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122805    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122808    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122811    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122815    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122818    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122820    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122823    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122826    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122828    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122831    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122834    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122837    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122839    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122842    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122844    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122847    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122850    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122852    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122855    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122859    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.124898 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122861    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122864    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122867    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122871    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122875    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122878    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122881    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122883    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122886    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122889    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122891    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122894    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122897    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122899    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122903    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122907    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122910    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122912    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122915    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.125416 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122918    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122921    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122923    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122926    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122929    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122931    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122933    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122936    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122939    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122942    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122944    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122948    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122950    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122953    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122956    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122959    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.125900 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.122962    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.122967    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123090    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123096    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123099    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123102    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123105    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123107    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123110    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123113    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123115    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123118    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123121    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123123    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:44.126305 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123126    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123132    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123136    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123139    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123142    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123145    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123148    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123150    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123153    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123156    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123158    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123161    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123164    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123167    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123169    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123173    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123175    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123178    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123181    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:44.126665 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123183    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123186    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123188    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123191    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123193    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123196    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123198    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123201    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123203    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123206    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123208    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123211    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123214    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123216    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123219    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123221    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123224    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123226    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123229    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123232    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:44.127149 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123234    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123237    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123239    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123242    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123244    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123247    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123250    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123252    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123255    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123258    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123261    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123264    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123266    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123269    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123272    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123274    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123277    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123279    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123282    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:44.127645 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123286    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123289    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123292    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123294    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123297    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123299    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123302    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123305    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123307    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123310    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123312    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123315    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123317    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123320    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123322    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123325    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123328    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123330    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123333    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:44.128107 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123336    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123339    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123342    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123346    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123349    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123352    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123355    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123358    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:44.123361    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.123366    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.123476    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.125303    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.126166    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:44.128585 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.126262    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:44.128958 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.126835    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:44.128958 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.128757    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:44.132882 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.132865    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:44.142138 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.142120    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:44.144961 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.144946    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:44.145843 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.145829    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:44.149818 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.149786    2581 fs.go:135] Filesystem UUIDs: map[12f07e5e-78a1-4bb6-8958-c268bb98c3eb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b00802ad-5673-4c22-9b9d-0f892a201c99:/dev/nvme0n1p4]
Sep 16 18:43:44.149896 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.149816    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:44.156174 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.156061    2581 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:44.15443423 +0000 UTC m=+0.305538458 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098290 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2814ea9cc091d8d7f1a0f34b873ce1 SystemUUID:ec2814ea-9cc0-91d8-d7f1-a0f34b873ce1 BootID:e23edaa5-7484-4bf5-9c9a-e0e154e749d5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:6c:a1:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:6c:a1:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:4f:d5:9f:f8:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:44.156174 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.156163    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:44.156304 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.156244    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:44.157373 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.157347    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:44.157511 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.157376    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:44.158437 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.158427    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:44.158471 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.158440    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:44.158471 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.158447    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:44.158947 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.158931    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:44.158983 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.158959    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:44.159756 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.159744    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:44.163912 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.163900    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:44.164022 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164013    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:44.164052 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164041    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:44.164052 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164051    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:44.164105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164062    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:44.164105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164071    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:44.164105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164079    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:44.164443 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164430    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:44.164496 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.164452    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:44.166600 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.166582    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:44.168619 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.168603    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:44.168695 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.168641    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:44.170375 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170360    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170381    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170406    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170414    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170426    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170434    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170453    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170461    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:44.170472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170470    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:44.170707 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170479    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:44.170707 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170492    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:44.170707 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.170501    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:44.172012 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.172002    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:44.172073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.172015    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:44.174129 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.174106    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:44.174129 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.174106    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:44.175961 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.175946    2581 server.go:1268] "Started kubelet"
Sep 16 18:43:44.176040 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.176009    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:44.176090 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.176044    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:44.176141 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.176115    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:44.176803 ip-10-0-130-142 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:44.177708 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.177680    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:44.180754 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.180707    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:44.180896 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.180884    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:44.182089 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.182070    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:44.182620 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.182594    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:44.182758 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.182738    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.182960 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.182940    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:44.183187 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183165    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:44.183187 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183171    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:44.183187 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183191    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:44.183364 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183281    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:44.183364 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183288    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:44.183826 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183806    2581 factory.go:55] Registering systemd factory
Sep 16 18:43:44.183826 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.183828    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:44.183936 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.182872    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-142.ec2.internal.18d5e15ec9a61a3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-142.ec2.internal,UID:ip-10-0-130-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-142.ec2.internal,},FirstTimestamp:2026-09-16 18:43:44.175921724 +0000 UTC m=+0.327025950,LastTimestamp:2026-09-16 18:43:44.175921724 +0000 UTC m=+0.327025950,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-142.ec2.internal,}"
Sep 16 18:43:44.184020 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184009    2581 factory.go:153] Registering CRI-O factory
Sep 16 18:43:44.184020 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184021    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:44.184126 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184072    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:44.184126 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184085    2581 factory.go:103] Registering Raw factory
Sep 16 18:43:44.184126 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184095    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:44.184412 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.184399    2581 manager.go:349] Starting recovery of all containers
Sep 16 18:43:44.190927 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.190761    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:43:44.191018 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.190933    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:43:44.194009 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.193984    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xzf9h"
Sep 16 18:43:44.196490 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.196472    2581 manager.go:354] Recovery completed
Sep 16 18:43:44.200586 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.200566    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xzf9h"
Sep 16 18:43:44.201407 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.201375    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.203167 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203150    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.203233 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203177    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.203233 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203189    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.203707 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203691    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:44.203707 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203706    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:44.203825 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203724    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:44.203825 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.203751    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.204539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.204523    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.204609 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.204557    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.204609 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.204571    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.205534 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.205455    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-142.ec2.internal.18d5e15ecb45d03e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-142.ec2.internal,UID:ip-10-0-130-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-142.ec2.internal,},FirstTimestamp:2026-09-16 18:43:44.203165758 +0000 UTC m=+0.354269984,LastTimestamp:2026-09-16 18:43:44.203165758 +0000 UTC m=+0.354269984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-142.ec2.internal,}"
Sep 16 18:43:44.205864 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.205853    2581 policy_none.go:49] "None policy: Start"
Sep 16 18:43:44.205910 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.205870    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:44.205910 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.205884    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:44.207948 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.207932    2581 policy_none.go:47] "Start"
Sep 16 18:43:44.230986 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.230973    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:44.231150 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.231017    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:44.231150 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231030    2581 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:44.231347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231334    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:44.231453 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231352    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:44.231778 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231762    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:44.231847 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231829    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:44.231847 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.231838    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:44.232437 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.232417    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:44.232514 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.232470    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.318259 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.318226    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:44.319401 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.319365    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:44.319495 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.319431    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:44.319495 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.319470    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:44.319570 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.319524    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:44.324316 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.324295    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:44.331706 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.331692    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.332558 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.332542    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.332624 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.332585    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.332624 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.332606    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.332699 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.332640    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.347429 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.347408    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.347514 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.347432    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-142.ec2.internal\": node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.369599 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.369530    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.420529 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.419752    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"]
Sep 16 18:43:44.420626 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.420574    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.421680 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.421657    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.421771 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.421691    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.421771 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.421702    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.423902 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.423888    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.424025 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.424060 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424041    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.424724 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424711    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.424783 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424735    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.424783 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424744    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.424849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424713    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.424849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424825    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.424849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.424839    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.427005 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.426991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.427069 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.427018    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:44.427766 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.427749    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:44.427845 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.427775    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:44.427845 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.427790    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:44.452937 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.452914    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-142.ec2.internal\" not found" node="ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.457253 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.457235    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-142.ec2.internal\" not found" node="ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.470211 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.470196    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.485028 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.485008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.485108 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.485039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.485108 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.485063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58d471a5ef85e5ea0c5b87f8f80839ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-142.ec2.internal\" (UID: \"58d471a5ef85e5ea0c5b87f8f80839ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.571017 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.570982    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.585595 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58d471a5ef85e5ea0c5b87f8f80839ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-142.ec2.internal\" (UID: \"58d471a5ef85e5ea0c5b87f8f80839ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.585739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.585739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.585739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.585739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58d471a5ef85e5ea0c5b87f8f80839ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-142.ec2.internal\" (UID: \"58d471a5ef85e5ea0c5b87f8f80839ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.585739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.585729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec086193683eccdbe4940053133614d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal\" (UID: \"7ec086193683eccdbe4940053133614d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.671997 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.671929    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:44.755873 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.755839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.759465 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:44.759440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:44.772532 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:44.772514    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:45.962208 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:45.962185    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.062735 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.062656    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.163353 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.163325    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.214786 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.214763    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:46.215181 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.214920    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:46.264235 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.264211    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.294526 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.294493    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:45 +0000 UTC" deadline="2028-03-02 04:06:53.208382001 +0000 UTC"
Sep 16 18:43:46.294526 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.294522    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12777h23m6.91386356s"
Sep 16 18:43:46.364863 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.364793    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.385892 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:46.385863    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec086193683eccdbe4940053133614d.slice/crio-bb3427f89fdabe679594d47f0e6ddcff00c25b32f9ca3adbd2808327351b201d WatchSource:0}: Error finding container bb3427f89fdabe679594d47f0e6ddcff00c25b32f9ca3adbd2808327351b201d: Status 404 returned error can't find the container with id bb3427f89fdabe679594d47f0e6ddcff00c25b32f9ca3adbd2808327351b201d
Sep 16 18:43:46.386284 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:46.386262    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d471a5ef85e5ea0c5b87f8f80839ec.slice/crio-d90183871ba65a1b1238eb2d6b33decfd7b4d3b3f65e9e65267d62cb89f6bed2 WatchSource:0}: Error finding container d90183871ba65a1b1238eb2d6b33decfd7b4d3b3f65e9e65267d62cb89f6bed2: Status 404 returned error can't find the container with id d90183871ba65a1b1238eb2d6b33decfd7b4d3b3f65e9e65267d62cb89f6bed2
Sep 16 18:43:46.391258 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.391241    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:46.412105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.412054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal" event={"ID":"7ec086193683eccdbe4940053133614d","Type":"ContainerStarted","Data":"bb3427f89fdabe679594d47f0e6ddcff00c25b32f9ca3adbd2808327351b201d"}
Sep 16 18:43:46.413027 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.413007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal" event={"ID":"58d471a5ef85e5ea0c5b87f8f80839ec","Type":"ContainerStarted","Data":"d90183871ba65a1b1238eb2d6b33decfd7b4d3b3f65e9e65267d62cb89f6bed2"}
Sep 16 18:43:46.465190 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.465167    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.539752 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.539727    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:46.565735 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.565704    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.666387 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.666318    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.720781 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.720751    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:46.766965 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:46.766940    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-142.ec2.internal\" not found"
Sep 16 18:43:46.773951 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.773928    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:46.872550 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.872520    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:46.884160 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.884120    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:46.885311 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.885290    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal"
Sep 16 18:43:46.895225 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:46.895199    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:47.253714 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.253688    2581 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:47.258870 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.258849    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:47.259268 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.259244    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xqtnz","kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-tv858","openshift-multus/network-metrics-daemon-n9drp","openshift-network-diagnostics/network-check-target-h7gn6","openshift-network-operator/iptables-alerter-8ttd9","openshift-ovn-kubernetes/ovnkube-node-knw5n","kube-system/konnectivity-agent-6nm7n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q","openshift-cluster-node-tuning-operator/tuned-5spw9","openshift-dns/node-resolver-gfnjj","openshift-image-registry/node-ca-bpd6c","openshift-multus/multus-n87v8"]
Sep 16 18:43:47.261680 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.261659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.263904 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.263881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.264261 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.264135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:47.264261 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.264178    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.264686 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.264564    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fw9kr\""
Sep 16 18:43:47.265350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.265326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.266772 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.266575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.267281 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.267214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.267371 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.267335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-65lsj\""
Sep 16 18:43:47.267688 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.267663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:47.267789 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.267730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.269511 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.269489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.269809 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.269694    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:47.269809 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.269752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.270002 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.269974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:47.270002 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.269990    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w5zp6\""
Sep 16 18:43:47.270128 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.270106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.271538 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.271389    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:47.272007 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.271666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.272007 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.271736    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:47.272121 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.272111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzhlt\""
Sep 16 18:43:47.272209 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.272131    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:47.272209 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.272189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:47.273861 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.273843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:47.273951 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.273911    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:47.276273 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.275999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.278439 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.278329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.278439 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.278390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:47.278439 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.278395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:47.278646 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.278494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8565d\""
Sep 16 18:43:47.278646 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.278549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:47.280723 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.280706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.283280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:47.283280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283106    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9lr5h\""
Sep 16 18:43:47.283280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.283280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283254    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:47.283651 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9pl\" (UniqueName: \"kubernetes.io/projected/1bf7b225-73e0-4208-a504-55399eb43561-kube-api-access-9b9pl\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.283740 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvghm\" (UniqueName: \"kubernetes.io/projected/f4f64602-4dbb-484f-ba94-3c722eb72730-kube-api-access-pvghm\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.283740 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283740 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-netns\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283740 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-multus\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-multus-daemon-config\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-multus-certs\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-system-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-k8s-cni-cncf-io\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.283937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-etc-kubernetes\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ghn\" (UniqueName: \"kubernetes.io/projected/24eaddca-b749-49b7-9641-3ef3070ad410-kube-api-access-95ghn\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.283993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-hostroot\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjjm\" (UniqueName: \"kubernetes.io/projected/b581ff8c-1c56-4015-89bf-b67d85e91e6b-kube-api-access-vzjjm\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2w25\" (UniqueName: \"kubernetes.io/projected/e1632371-d9cd-4766-a6c0-521a31af4fe6-kube-api-access-f2w25\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf7b225-73e0-4208-a504-55399eb43561-host-slash\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-cnibin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cnibin\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-cni-binary-copy\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f64602-4dbb-484f-ba94-3c722eb72730-host\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f64602-4dbb-484f-ba94-3c722eb72730-serviceca\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-socket-dir-parent\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-bin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-kubelet\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-conf-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284579    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.284936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.284640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1bf7b225-73e0-4208-a504-55399eb43561-iptables-alerter-script\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.285469 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.285316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:47.285702 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.285680    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.285796 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.285763    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.285925 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.285903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.286408 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.286390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qr82x\""
Sep 16 18:43:47.287849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.287808    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.287849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.287844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vqlh\""
Sep 16 18:43:47.288054 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.288033    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.288833 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.288595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.290524 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.290488    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:47.290616 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.290570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lcd92\""
Sep 16 18:43:47.290782 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.290761    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:47.325423 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.325401    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qx5j9"
Sep 16 18:43:47.336176 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.336139    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qx5j9"
Sep 16 18:43:47.373033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.373009    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:47.385105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-lib-modules\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.385228 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s62f\" (UniqueName: \"kubernetes.io/projected/c1f334d5-be49-44c6-900b-baa0db9f3378-kube-api-access-8s62f\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.385228 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-slash\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385228 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-netns\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385228 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385228 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f64602-4dbb-484f-ba94-3c722eb72730-serviceca\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-socket-dir-parent\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-sys-fs\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2d54608-aad8-4ae1-a382-29808406a431-agent-certs\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-socket-dir-parent\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.385447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-tmp\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-bin\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-netd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-env-overrides\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-systemd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-conf\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvghm\" (UniqueName: \"kubernetes.io/projected/f4f64602-4dbb-484f-ba94-3c722eb72730-kube-api-access-pvghm\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-multus\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.385694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-multus-certs\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-etc-kubernetes\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f64602-4dbb-484f-ba94-3c722eb72730-serviceca\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ghn\" (UniqueName: \"kubernetes.io/projected/24eaddca-b749-49b7-9641-3ef3070ad410-kube-api-access-95ghn\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-multus\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-socket-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-etc-kubernetes\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-system-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-multus-certs\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2w25\" (UniqueName: \"kubernetes.io/projected/e1632371-d9cd-4766-a6c0-521a31af4fe6-kube-api-access-f2w25\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-run\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-system-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysconfig\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-kubelet\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.386029    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:47.386065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjjm\" (UniqueName: \"kubernetes.io/projected/b581ff8c-1c56-4015-89bf-b67d85e91e6b-kube-api-access-vzjjm\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.385964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-system-cni-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-ovn\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.386119    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:47.88607865 +0000 UTC m=+2.948417886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf7b225-73e0-4208-a504-55399eb43561-host-slash\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2d54608-aad8-4ae1-a382-29808406a431-konnectivity-ca\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf7b225-73e0-4208-a504-55399eb43561-host-slash\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-etc-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-log-socket\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7m9\" (UniqueName: \"kubernetes.io/projected/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-kube-api-access-pn7m9\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-dbus\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cnibin\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cnibin\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.386883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-modprobe-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-systemd\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-systemd-units\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-cni-binary-copy\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f64602-4dbb-484f-ba94-3c722eb72730-host\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-sys\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-host\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990619d1-4ef1-4d44-943f-b7236ff5c472-tmp-dir\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-bin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-kubelet\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-conf-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mhh\" (UniqueName: \"kubernetes.io/projected/990619d1-4ef1-4d44-943f-b7236ff5c472-kube-api-access-q8mhh\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f64602-4dbb-484f-ba94-3c722eb72730-host\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-var-lib-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.387486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-kubelet-config\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.386981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1bf7b225-73e0-4208-a504-55399eb43561-iptables-alerter-script\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-var-lib-kubelet\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-tuned\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9pl\" (UniqueName: \"kubernetes.io/projected/1bf7b225-73e0-4208-a504-55399eb43561-kube-api-access-9b9pl\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-netns\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-multus-daemon-config\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-cni-binary-copy\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24eaddca-b749-49b7-9641-3ef3070ad410-multus-daemon-config\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-cni-bin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-var-lib-kubelet\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-conf-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1bf7b225-73e0-4208-a504-55399eb43561-iptables-alerter-script\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-registration-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4ng\" (UniqueName: \"kubernetes.io/projected/96922f78-dfad-492f-aecf-bab191c1a66e-kube-api-access-wb4ng\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.388055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-kubernetes\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-k8s-cni-cncf-io\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/990619d1-4ef1-4d44-943f-b7236ff5c472-hosts-file\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-config\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-script-lib\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-hostroot\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-device-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-cnibin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.387999    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-netns\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-node-log\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-hostroot\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-host-run-k8s-cni-cncf-io\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-multus-cni-dir\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b581ff8c-1c56-4015-89bf-b67d85e91e6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.388637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24eaddca-b749-49b7-9641-3ef3070ad410-cnibin\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.389207 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.388479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b581ff8c-1c56-4015-89bf-b67d85e91e6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.400698 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.400633    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:47.400698 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.400659    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:47.400698 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.400674    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:47.400904 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.400746    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:47.900730371 +0000 UTC m=+2.963069617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:47.401054 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.401035    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:47.405515 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.405491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2w25\" (UniqueName: \"kubernetes.io/projected/e1632371-d9cd-4766-a6c0-521a31af4fe6-kube-api-access-f2w25\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.405515 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.405506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvghm\" (UniqueName: \"kubernetes.io/projected/f4f64602-4dbb-484f-ba94-3c722eb72730-kube-api-access-pvghm\") pod \"node-ca-bpd6c\" (UID: \"f4f64602-4dbb-484f-ba94-3c722eb72730\") " pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.405640 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.405498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ghn\" (UniqueName: \"kubernetes.io/projected/24eaddca-b749-49b7-9641-3ef3070ad410-kube-api-access-95ghn\") pod \"multus-n87v8\" (UID: \"24eaddca-b749-49b7-9641-3ef3070ad410\") " pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.405640 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.405498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjjm\" (UniqueName: \"kubernetes.io/projected/b581ff8c-1c56-4015-89bf-b67d85e91e6b-kube-api-access-vzjjm\") pod \"multus-additional-cni-plugins-tv858\" (UID: \"b581ff8c-1c56-4015-89bf-b67d85e91e6b\") " pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.406059 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.406041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9pl\" (UniqueName: \"kubernetes.io/projected/1bf7b225-73e0-4208-a504-55399eb43561-kube-api-access-9b9pl\") pod \"iptables-alerter-8ttd9\" (UID: \"1bf7b225-73e0-4208-a504-55399eb43561\") " pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.488689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-sys\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-host\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990619d1-4ef1-4d44-943f-b7236ff5c472-tmp-dir\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-sys\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mhh\" (UniqueName: \"kubernetes.io/projected/990619d1-4ef1-4d44-943f-b7236ff5c472-kube-api-access-q8mhh\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-host\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.488863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-var-lib-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-kubelet-config\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-var-lib-kubelet\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-tuned\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-registration-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-kubelet-config\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-var-lib-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.488991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4ng\" (UniqueName: \"kubernetes.io/projected/96922f78-dfad-492f-aecf-bab191c1a66e-kube-api-access-wb4ng\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-kubernetes\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-var-lib-kubelet\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/990619d1-4ef1-4d44-943f-b7236ff5c472-hosts-file\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990619d1-4ef1-4d44-943f-b7236ff5c472-tmp-dir\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-config\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-kubernetes\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/990619d1-4ef1-4d44-943f-b7236ff5c472-hosts-file\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-registration-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.489234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-script-lib\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-device-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-node-log\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-device-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-node-log\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-lib-modules\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s62f\" (UniqueName: \"kubernetes.io/projected/c1f334d5-be49-44c6-900b-baa0db9f3378-kube-api-access-8s62f\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-slash\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-lib-modules\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-slash\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-netns\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-config\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490033 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-sys-fs\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-netns\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2d54608-aad8-4ae1-a382-29808406a431-agent-certs\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-tmp\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-bin\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-sys-fs\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-netd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovnkube-script-lib\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-bin\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-env-overrides\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-cni-netd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-systemd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-systemd\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-conf\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.490838 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.489996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-socket-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-run\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysconfig\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-kubelet\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-ovn\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysctl-conf\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2d54608-aad8-4ae1-a382-29808406a431-konnectivity-ca\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96922f78-dfad-492f-aecf-bab191c1a66e-socket-dir\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-sysconfig\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-host-kubelet\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-run\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-run-ovn\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-etc-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-env-overrides\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-log-socket\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-etc-openvswitch\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.491614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-log-socket\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7m9\" (UniqueName: \"kubernetes.io/projected/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-kube-api-access-pn7m9\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-dbus\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-modprobe-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-systemd\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9528a95a-967e-48e3-9caf-5604a16ffa0c-dbus\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-systemd-units\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-modprobe-d\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2d54608-aad8-4ae1-a382-29808406a431-konnectivity-ca\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-systemd-units\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.490819    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.490872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-systemd\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.490924    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:43:47.990891246 +0000 UTC m=+3.053230490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:47.492379 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.491716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-etc-tuned\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492887 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.492523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2d54608-aad8-4ae1-a382-29808406a431-agent-certs\") pod \"konnectivity-agent-6nm7n\" (UID: \"d2d54608-aad8-4ae1-a382-29808406a431\") " pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.492887 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.492584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1f334d5-be49-44c6-900b-baa0db9f3378-tmp\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.492987 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.492888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.497543 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.497520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4ng\" (UniqueName: \"kubernetes.io/projected/96922f78-dfad-492f-aecf-bab191c1a66e-kube-api-access-wb4ng\") pod \"aws-ebs-csi-driver-node-4kg7q\" (UID: \"96922f78-dfad-492f-aecf-bab191c1a66e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.498639 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.498617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s62f\" (UniqueName: \"kubernetes.io/projected/c1f334d5-be49-44c6-900b-baa0db9f3378-kube-api-access-8s62f\") pod \"tuned-5spw9\" (UID: \"c1f334d5-be49-44c6-900b-baa0db9f3378\") " pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.498885 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.498867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mhh\" (UniqueName: \"kubernetes.io/projected/990619d1-4ef1-4d44-943f-b7236ff5c472-kube-api-access-q8mhh\") pod \"node-resolver-gfnjj\" (UID: \"990619d1-4ef1-4d44-943f-b7236ff5c472\") " pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.500724 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.500698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7m9\" (UniqueName: \"kubernetes.io/projected/65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c-kube-api-access-pn7m9\") pod \"ovnkube-node-knw5n\" (UID: \"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.576982 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.576906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ttd9"
Sep 16 18:43:47.583761 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.583738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bpd6c"
Sep 16 18:43:47.593084 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.593053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n87v8"
Sep 16 18:43:47.599190 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.599171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tv858"
Sep 16 18:43:47.605854 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.605833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:43:47.612059 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.612043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:43:47.618876 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.618859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q"
Sep 16 18:43:47.624604 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.624579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5spw9"
Sep 16 18:43:47.629380 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.629361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfnjj"
Sep 16 18:43:47.716594 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.716565    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:47.892426 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.892350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:47.892565 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.892504    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:47.892565 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.892561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:48.892545536 +0000 UTC m=+3.954884766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:47.992993 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.992962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:47.992993 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:47.992997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993113    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993124    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993144    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993173    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993198    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:43:48.993180119 +0000 UTC m=+4.055519348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:47.993248 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:47.993219    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:48.993209798 +0000 UTC m=+4.055549031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:48.206954 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.206925    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24eaddca_b749_49b7_9641_3ef3070ad410.slice/crio-04eccd56f246a0d3aedb3f9b389ea47eea7a21e5c2ef906f69ea7069912e4007 WatchSource:0}: Error finding container 04eccd56f246a0d3aedb3f9b389ea47eea7a21e5c2ef906f69ea7069912e4007: Status 404 returned error can't find the container with id 04eccd56f246a0d3aedb3f9b389ea47eea7a21e5c2ef906f69ea7069912e4007
Sep 16 18:43:48.207955 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.207894    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf7b225_73e0_4208_a504_55399eb43561.slice/crio-de48950c72742eb905b2c7617fee34c31d9e63872a1716bab7754090bcbc363e WatchSource:0}: Error finding container de48950c72742eb905b2c7617fee34c31d9e63872a1716bab7754090bcbc363e: Status 404 returned error can't find the container with id de48950c72742eb905b2c7617fee34c31d9e63872a1716bab7754090bcbc363e
Sep 16 18:43:48.209250 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.209225    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb581ff8c_1c56_4015_89bf_b67d85e91e6b.slice/crio-998a9989cf248b87b5c748bd79a1ad29633e43f185bba068d360235511aeb3d1 WatchSource:0}: Error finding container 998a9989cf248b87b5c748bd79a1ad29633e43f185bba068d360235511aeb3d1: Status 404 returned error can't find the container with id 998a9989cf248b87b5c748bd79a1ad29633e43f185bba068d360235511aeb3d1
Sep 16 18:43:48.209581 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.209545    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f64602_4dbb_484f_ba94_3c722eb72730.slice/crio-667be6c867673eb6c709956d2bc07d4e38dd0966944a14b22694dd2f2673cd04 WatchSource:0}: Error finding container 667be6c867673eb6c709956d2bc07d4e38dd0966944a14b22694dd2f2673cd04: Status 404 returned error can't find the container with id 667be6c867673eb6c709956d2bc07d4e38dd0966944a14b22694dd2f2673cd04
Sep 16 18:43:48.210320 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.210288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96922f78_dfad_492f_aecf_bab191c1a66e.slice/crio-1224db7927fedb59e70cea857f7ed19c08d716503a26050025eb4fa807bbe750 WatchSource:0}: Error finding container 1224db7927fedb59e70cea857f7ed19c08d716503a26050025eb4fa807bbe750: Status 404 returned error can't find the container with id 1224db7927fedb59e70cea857f7ed19c08d716503a26050025eb4fa807bbe750
Sep 16 18:43:48.212864 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.212842    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f3ab43_1d3e_4c70_a727_2ce2f5a67f0c.slice/crio-e598b0bbc3b7567caaeaf5a7efa7a677307e73a7ba5b0637d16b46131e4f6785 WatchSource:0}: Error finding container e598b0bbc3b7567caaeaf5a7efa7a677307e73a7ba5b0637d16b46131e4f6785: Status 404 returned error can't find the container with id e598b0bbc3b7567caaeaf5a7efa7a677307e73a7ba5b0637d16b46131e4f6785
Sep 16 18:43:48.213859 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.213835    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f334d5_be49_44c6_900b_baa0db9f3378.slice/crio-8decf087b060b8dbe566a326f9c7a757000cb8cfd5e6188c32a4e60cb4df0edb WatchSource:0}: Error finding container 8decf087b060b8dbe566a326f9c7a757000cb8cfd5e6188c32a4e60cb4df0edb: Status 404 returned error can't find the container with id 8decf087b060b8dbe566a326f9c7a757000cb8cfd5e6188c32a4e60cb4df0edb
Sep 16 18:43:48.215269 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.215100    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d54608_aad8_4ae1_a382_29808406a431.slice/crio-ddbadd88aaff3f11dd6b2bb03106adefea5afd05002aa91ca0727638f49a99b8 WatchSource:0}: Error finding container ddbadd88aaff3f11dd6b2bb03106adefea5afd05002aa91ca0727638f49a99b8: Status 404 returned error can't find the container with id ddbadd88aaff3f11dd6b2bb03106adefea5afd05002aa91ca0727638f49a99b8
Sep 16 18:43:48.216964 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:43:48.216644    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990619d1_4ef1_4d44_943f_b7236ff5c472.slice/crio-038b22595fbd6d847cfb5d41f5cf44511e9e88a5ebbc4a448f1f06a4d5786f34 WatchSource:0}: Error finding container 038b22595fbd6d847cfb5d41f5cf44511e9e88a5ebbc4a448f1f06a4d5786f34: Status 404 returned error can't find the container with id 038b22595fbd6d847cfb5d41f5cf44511e9e88a5ebbc4a448f1f06a4d5786f34
Sep 16 18:43:48.339316 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.339285    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:47 +0000 UTC" deadline="2028-03-04 04:46:22.021578438 +0000 UTC"
Sep 16 18:43:48.339316 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.339314    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12826h2m33.682268344s"
Sep 16 18:43:48.408908 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.408880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:48.409052 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:48.409027    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:48.419211 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.419184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal" event={"ID":"58d471a5ef85e5ea0c5b87f8f80839ec","Type":"ContainerStarted","Data":"d92cd653416195ec42f00f03fd9fa930cd7bf2433a4401b7ddba1f071a083cd0"}
Sep 16 18:43:48.420359 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.420330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfnjj" event={"ID":"990619d1-4ef1-4d44-943f-b7236ff5c472","Type":"ContainerStarted","Data":"038b22595fbd6d847cfb5d41f5cf44511e9e88a5ebbc4a448f1f06a4d5786f34"}
Sep 16 18:43:48.421528 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.421509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"e598b0bbc3b7567caaeaf5a7efa7a677307e73a7ba5b0637d16b46131e4f6785"}
Sep 16 18:43:48.422725 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.422703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ttd9" event={"ID":"1bf7b225-73e0-4208-a504-55399eb43561","Type":"ContainerStarted","Data":"de48950c72742eb905b2c7617fee34c31d9e63872a1716bab7754090bcbc363e"}
Sep 16 18:43:48.423805 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.423785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n87v8" event={"ID":"24eaddca-b749-49b7-9641-3ef3070ad410","Type":"ContainerStarted","Data":"04eccd56f246a0d3aedb3f9b389ea47eea7a21e5c2ef906f69ea7069912e4007"}
Sep 16 18:43:48.424893 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.424873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6nm7n" event={"ID":"d2d54608-aad8-4ae1-a382-29808406a431","Type":"ContainerStarted","Data":"ddbadd88aaff3f11dd6b2bb03106adefea5afd05002aa91ca0727638f49a99b8"}
Sep 16 18:43:48.425894 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.425874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5spw9" event={"ID":"c1f334d5-be49-44c6-900b-baa0db9f3378","Type":"ContainerStarted","Data":"8decf087b060b8dbe566a326f9c7a757000cb8cfd5e6188c32a4e60cb4df0edb"}
Sep 16 18:43:48.426952 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.426926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q" event={"ID":"96922f78-dfad-492f-aecf-bab191c1a66e","Type":"ContainerStarted","Data":"1224db7927fedb59e70cea857f7ed19c08d716503a26050025eb4fa807bbe750"}
Sep 16 18:43:48.428105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.428087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bpd6c" event={"ID":"f4f64602-4dbb-484f-ba94-3c722eb72730","Type":"ContainerStarted","Data":"667be6c867673eb6c709956d2bc07d4e38dd0966944a14b22694dd2f2673cd04"}
Sep 16 18:43:48.429160 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.429128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerStarted","Data":"998a9989cf248b87b5c748bd79a1ad29633e43f185bba068d360235511aeb3d1"}
Sep 16 18:43:48.901401 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:48.900789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:48.901401 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:48.900938    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:48.901401 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:48.901001    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:50.900981596 +0000 UTC m=+5.963320831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.001943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.001996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002169    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002187    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002199    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.002238756 +0000 UTC m=+6.064578007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002642    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:49.002728 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.002692    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.00267701 +0000 UTC m=+6.065016245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:49.340299 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.340184    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:47 +0000 UTC" deadline="2028-06-04 19:13:20.421318786 +0000 UTC"
Sep 16 18:43:49.340299 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.340242    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15048h29m31.081080855s"
Sep 16 18:43:49.410297 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.410268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:49.410472 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.410392    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:49.412532 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.412496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:49.412657 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:49.412633    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:49.437683 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.437652    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ec086193683eccdbe4940053133614d" containerID="bdd3b33aaf45265260e35942a8885147b2167c7c69b34af120e25d722d781bbb" exitCode=0
Sep 16 18:43:49.438554 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.438529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal" event={"ID":"7ec086193683eccdbe4940053133614d","Type":"ContainerDied","Data":"bdd3b33aaf45265260e35942a8885147b2167c7c69b34af120e25d722d781bbb"}
Sep 16 18:43:49.463698 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:49.462747    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-142.ec2.internal" podStartSLOduration=3.462730074 podStartE2EDuration="3.462730074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:48.43638323 +0000 UTC m=+3.498722478" watchObservedRunningTime="2026-09-16 18:43:49.462730074 +0000 UTC m=+4.525069327"
Sep 16 18:43:50.409841 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:50.409236    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:50.409841 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:50.409372    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:50.481214 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:50.481177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal" event={"ID":"7ec086193683eccdbe4940053133614d","Type":"ContainerStarted","Data":"46fde4751f779d5ca69727fd9a92d5ae0ceac3a8bc9f8f659102ff99f81e4b5f"}
Sep 16 18:43:50.917786 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:50.917191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:50.917786 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:50.917344    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:50.917786 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:50.917406    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.917386812 +0000 UTC m=+9.979726047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:51.018097 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:51.017850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:51.018097 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:51.017909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:51.018097 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018059    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:51.018381 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018143    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:43:55.018123681 +0000 UTC m=+10.080463127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:51.018442 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018069    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:51.018442 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018420    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:51.018442 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018435    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:51.018596 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.018494    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:55.018478648 +0000 UTC m=+10.080817891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:51.409365 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:51.409283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:51.409520 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.409414    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:51.409625 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:51.409601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:51.409760 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:51.409721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:52.409590 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:52.409036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:52.409590 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:52.409192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:53.409496 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:53.409464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:53.409699 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:53.409514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:53.409699 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:53.409643    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:53.410048 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:53.409810    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:54.409334 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:54.409204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:54.409525 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:54.409355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:54.951306 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:54.951267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:54.951768 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:54.951488    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:54.951768 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:54.951568    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:02.951545688 +0000 UTC m=+18.013884922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:55.051689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:55.051646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:55.051689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:55.051693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:55.051923 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.051888    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:55.051923 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.051910    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:55.051923 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.051922    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:55.052073 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.051932    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:55.052073 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.051981    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:03.05196375 +0000 UTC m=+18.114302988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:55.052264 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.052108    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:03.052090676 +0000 UTC m=+18.114429916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:55.410916 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:55.410192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:55.410916 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:55.410209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:55.410916 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.410570    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:55.410916 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:55.410632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:56.409031 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:56.408989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:56.409517 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:56.409165    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:57.409082 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:57.409036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:57.409588 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:57.409036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:57.409588 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:57.409174    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:57.409588 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:57.409216    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:43:58.409145 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:58.409119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:43:58.409439 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:58.409227    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:43:59.409424 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:59.409394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:43:59.409833 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:59.409486    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:43:59.409833 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:43:59.409557    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:43:59.409833 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:43:59.409654    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:00.408906 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:00.408863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:00.409088 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:00.408982    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:01.409269 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:01.409230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:01.409703 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:01.409244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:01.409703 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:01.409354    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:01.409703 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:01.409472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:02.409291 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:02.409256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:02.409772 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:02.409389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:03.007607 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:03.007575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:03.007776 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.007730    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:03.007825 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.007794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.007778688 +0000 UTC m=+34.070117916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:03.108464 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:03.108425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:03.108640 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:03.108474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:03.108640 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108590    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:03.108640 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108637    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:03.108790 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108658    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:03.108790 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108661    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.108642445 +0000 UTC m=+34.170981677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:03.108790 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108667    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:03.108790 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.108709    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.108694619 +0000 UTC m=+34.171033862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:03.408960 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:03.408923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:03.409182 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:03.408923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:03.409182 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.409036    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:03.409182 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:03.409130    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:04.408643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:04.408613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:04.409096 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:04.408736    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:05.409575 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:05.409535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:05.410025 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:05.409659    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:05.410025 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:05.409733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:05.410025 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:05.409849    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:06.409432 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.409266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:06.409571 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:06.409516    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:06.514669 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.514634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5spw9" event={"ID":"c1f334d5-be49-44c6-900b-baa0db9f3378","Type":"ContainerStarted","Data":"1be0e7637e68cfda67e1d66afd36de2c6d5dffd726fe58dca4d25c50bf2774ae"}
Sep 16 18:44:06.516275 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.516248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q" event={"ID":"96922f78-dfad-492f-aecf-bab191c1a66e","Type":"ContainerStarted","Data":"4c25de892d9baf37bec297029109c6b0a7cc48cd207c6ca6c585ed532468f551"}
Sep 16 18:44:06.517700 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.517661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bpd6c" event={"ID":"f4f64602-4dbb-484f-ba94-3c722eb72730","Type":"ContainerStarted","Data":"bc5783e372b4199fae7afd326791e140056ce92dee3a797bcd4d5aa38f456df1"}
Sep 16 18:44:06.519131 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.519108    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="841e16d550326471bd70bc7ad12e1084e25fdf7e76ae26c723a8f7f991cd7355" exitCode=0
Sep 16 18:44:06.519265 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.519177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"841e16d550326471bd70bc7ad12e1084e25fdf7e76ae26c723a8f7f991cd7355"}
Sep 16 18:44:06.520656 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.520615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfnjj" event={"ID":"990619d1-4ef1-4d44-943f-b7236ff5c472","Type":"ContainerStarted","Data":"2f8bdb99b3175355211537f9a56cfad827825f4457d31302c1e1db5604383043"}
Sep 16 18:44:06.522844 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.522817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"0ecfc234878575951090e9f05f5d36411267f12beffa20d503b1fb8d7fd95b5c"}
Sep 16 18:44:06.522933 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.522892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"3b8056e05d0e2e01a3444c2eed94031d9ea29212a7d426995835aaffd1cb7a88"}
Sep 16 18:44:06.522988 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.522942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"4720af4d12130775a9ab78bc4451feb5d7206c498782b1ef10bbadbb90a89c30"}
Sep 16 18:44:06.523053 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.522991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"03247e2e7540a83b17d492307109d77effcbcc7b34ec1c8f29d6e4d1fbd9f9f8"}
Sep 16 18:44:06.524350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.524320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n87v8" event={"ID":"24eaddca-b749-49b7-9641-3ef3070ad410","Type":"ContainerStarted","Data":"acbe6f3f20b31e95eddad2ef6cc02f433b968a1a6117026e58fd9d5f7f80a570"}
Sep 16 18:44:06.525635 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.525616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6nm7n" event={"ID":"d2d54608-aad8-4ae1-a382-29808406a431","Type":"ContainerStarted","Data":"c26cc7ead20f37eb84eb68124e47cb6d1741e1ff50b2f021ccc50b4cce826436"}
Sep 16 18:44:06.531475 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.531430    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5spw9" podStartSLOduration=3.7164589599999998 podStartE2EDuration="21.531399202s" totalImagesPullingTime="17.814940242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:06.531210099 +0000 UTC m=+21.593549347" watchObservedRunningTime="2026-09-16 18:44:06.531399202 +0000 UTC m=+21.593738435"
Sep 16 18:44:06.531753 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.531733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-142.ec2.internal" podStartSLOduration=20.531725606 podStartE2EDuration="20.531725606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:50.499586295 +0000 UTC m=+5.561925547" watchObservedRunningTime="2026-09-16 18:44:06.531725606 +0000 UTC m=+21.594064857"
Sep 16 18:44:06.547493 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.547418    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n87v8" podStartSLOduration=3.619352417 podStartE2EDuration="21.547401943s" totalImagesPullingTime="17.928049526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:06.546944564 +0000 UTC m=+21.609283815" watchObservedRunningTime="2026-09-16 18:44:06.547401943 +0000 UTC m=+21.609741194"
Sep 16 18:44:06.560927 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.560871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gfnjj" podStartSLOduration=3.749911187 podStartE2EDuration="21.560854626s" totalImagesPullingTime="17.810943439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:06.560441963 +0000 UTC m=+21.622781214" watchObservedRunningTime="2026-09-16 18:44:06.560854626 +0000 UTC m=+21.623193876"
Sep 16 18:44:06.593271 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.593223    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6nm7n" podStartSLOduration=11.398466264 podStartE2EDuration="21.593204886s" totalImagesPullingTime="10.194738622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:06.592574842 +0000 UTC m=+21.654914093" watchObservedRunningTime="2026-09-16 18:44:06.593204886 +0000 UTC m=+21.655544137"
Sep 16 18:44:06.613128 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:06.613084    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bpd6c" podStartSLOduration=3.795295934 podStartE2EDuration="21.61306565s" totalImagesPullingTime="17.817769716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:06.612545549 +0000 UTC m=+21.674884801" watchObservedRunningTime="2026-09-16 18:44:06.61306565 +0000 UTC m=+21.675404900"
Sep 16 18:44:07.205717 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.205686    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:07.335434 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.335247    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:07.205713177Z","UUID":"0ff43629-f627-4e1b-80ba-25add96f97f9","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:07.336951 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.336923    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:07.337074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.336960    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:07.408507 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.408479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:07.408719 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.408484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:07.408719 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:07.408598    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:07.408719 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:07.408702    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:07.530142 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.530099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ttd9" event={"ID":"1bf7b225-73e0-4208-a504-55399eb43561","Type":"ContainerStarted","Data":"9b6e975673e8421dffe413069c7ef8954bfbfefd3ee2f6a31a793c6370e16ab7"}
Sep 16 18:44:07.535617 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.535575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q" event={"ID":"96922f78-dfad-492f-aecf-bab191c1a66e","Type":"ContainerStarted","Data":"a082160113889cd0d55892e389542de1d4677a1b4f10728574e0899fd11d4940"}
Sep 16 18:44:07.539291 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.539251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"33efefe22efc9493806f3e6737aadfc0a2c19a9bd014b12b26a8d0bb9116e688"}
Sep 16 18:44:07.539439 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.539385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"90081bcfa802b600988093593f7b3cad1fea92ba79366ef4c34399fdea396dcf"}
Sep 16 18:44:07.545184 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:07.545125    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8ttd9" podStartSLOduration=4.72651764 podStartE2EDuration="22.54511081s" totalImagesPullingTime="17.81859317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:07.545094735 +0000 UTC m=+22.607433989" watchObservedRunningTime="2026-09-16 18:44:07.54511081 +0000 UTC m=+22.607450058"
Sep 16 18:44:08.408672 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:08.408636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:08.408870 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:08.408766    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:08.544526 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:08.544492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q" event={"ID":"96922f78-dfad-492f-aecf-bab191c1a66e","Type":"ContainerStarted","Data":"b740f919e8982437e9df74c6db33205eba28d56474d73fcb40b3e8d191558615"}
Sep 16 18:44:09.408505 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.408467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:09.408809 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:09.408608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:09.408809 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.408658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:09.408809 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:09.408789    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:09.550371 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.550334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"1be743ca814c586376126dcbe2eb5ded907542023d86e538bc0f73630a224676"}
Sep 16 18:44:09.790910 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.790829    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:44:09.791559 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.791532    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:44:09.806898 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:09.806795    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4kg7q" podStartSLOduration=5.424128897 podStartE2EDuration="24.806778502s" totalImagesPullingTime="19.382649605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:08.560937439 +0000 UTC m=+23.623276692" watchObservedRunningTime="2026-09-16 18:44:09.806778502 +0000 UTC m=+24.869117753"
Sep 16 18:44:10.408754 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:10.408720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:10.408926 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:10.408825    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:10.552513 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:10.552478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:44:10.552955 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:10.552938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6nm7n"
Sep 16 18:44:11.409551 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:11.409376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:11.409781 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:11.409453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:11.409781 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:11.409616    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:11.409781 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:11.409742    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:11.556068 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:11.556034    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="7955e4981b9a3b4c5ede2c048833612301c07560cb751a338cc213590441ecb5" exitCode=0
Sep 16 18:44:11.556604 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:11.556111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"7955e4981b9a3b4c5ede2c048833612301c07560cb751a338cc213590441ecb5"}
Sep 16 18:44:11.560121 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:11.560096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" event={"ID":"65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c","Type":"ContainerStarted","Data":"1c0e87ea854bda11c63d3ed81613a069e6262c38d6d774c8435a434f0ea4f047"}
Sep 16 18:44:12.409353 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.409310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:12.409490 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:12.409413    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:12.564062 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.563985    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="3497695a33c1317b7504a80e87016cc2c7b6393c972e3b6778f395b2576df61e" exitCode=0
Sep 16 18:44:12.564062 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.564027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"3497695a33c1317b7504a80e87016cc2c7b6393c972e3b6778f395b2576df61e"}
Sep 16 18:44:12.564784 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.564454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:12.564784 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.564514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:12.564784 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.564560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:12.580576 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.580551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:12.580774 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.580756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:12.589142 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.589094    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n" podStartSLOduration=9.758661915 podStartE2EDuration="27.589077048s" totalImagesPullingTime="17.830415133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:11.627324361 +0000 UTC m=+26.689663612" watchObservedRunningTime="2026-09-16 18:44:12.589077048 +0000 UTC m=+27.651416300"
Sep 16 18:44:12.896593 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.896390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqtnz"]
Sep 16 18:44:12.896918 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.896896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:12.897013 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:12.896992    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:12.899780 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.899659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9drp"]
Sep 16 18:44:12.900283 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.900128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:12.900375 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:12.900325    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:12.901359 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.901323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7gn6"]
Sep 16 18:44:12.901472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:12.901423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:12.901532 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:12.901494    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:13.568989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:13.568895    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="cd8622fa68b8c1a116c58df2f341a466cf0ad8cb99080aa3d0e06eef77bad83a" exitCode=0
Sep 16 18:44:13.568989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:13.568960    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"cd8622fa68b8c1a116c58df2f341a466cf0ad8cb99080aa3d0e06eef77bad83a"}
Sep 16 18:44:14.409125 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:14.409092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:14.409125 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:14.409119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:14.409361 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:14.409092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:14.409361 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:14.409249    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:14.409361 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:14.409309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:14.409492 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:14.409395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:16.375324 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:16.375295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfnjj_990619d1-4ef1-4d44-943f-b7236ff5c472/dns-node-resolver/0.log"
Sep 16 18:44:16.409470 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:16.409439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:16.409633 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:16.409452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:16.409633 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:16.409555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:16.409633 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:16.409453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:16.409633 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:16.409625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:16.409842 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:16.409706    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:17.350322 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:17.350291    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bpd6c_f4f64602-4dbb-484f-ba94-3c722eb72730/node-ca/0.log"
Sep 16 18:44:18.408739 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.408498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:18.409131 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.408553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:18.409131 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:18.408837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqtnz" podUID="9528a95a-967e-48e3-9caf-5604a16ffa0c"
Sep 16 18:44:18.409131 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.408553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:18.409131 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:18.408922    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9drp" podUID="e1632371-d9cd-4766-a6c0-521a31af4fe6"
Sep 16 18:44:18.409131 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:18.408992    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7gn6" podUID="46c18fd8-6738-4435-a7dc-6180ef83b6dd"
Sep 16 18:44:18.586225 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.586188    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="f52be76065171ee5f17bf6ffd7015156de8f990a113e6d84fd2765ec263e4c55" exitCode=0
Sep 16 18:44:18.586399 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.586238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"f52be76065171ee5f17bf6ffd7015156de8f990a113e6d84fd2765ec263e4c55"}
Sep 16 18:44:18.714214 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.714188    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-142.ec2.internal" event="NodeReady"
Sep 16 18:44:18.714341 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.714316    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:44:18.765008 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.764979    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tdz6v"]
Sep 16 18:44:18.778533 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.778504    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s55fk"]
Sep 16 18:44:18.778665 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.778644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:18.780994 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.780973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nvn9q\""
Sep 16 18:44:18.781298 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.781277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:44:18.781416 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.781330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:44:18.781548 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.781536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:44:18.792721 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.792701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tdz6v"]
Sep 16 18:44:18.792813 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.792730    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6pt88"]
Sep 16 18:44:18.792862 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.792833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:18.795350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.795328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:44:18.795350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.795340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:44:18.795518 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.795391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:44:18.795518 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.795344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlsxc\""
Sep 16 18:44:18.795518 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.795476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:44:18.806849 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.806828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s55fk"]
Sep 16 18:44:18.806937 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.806863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pt88"]
Sep 16 18:44:18.806975 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.806938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:18.809097 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.809070    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:44:18.809097 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.809078    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6mpm2\""
Sep 16 18:44:18.809303 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.809291    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:44:18.930559 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/519727fa-2299-44e2-a7bd-65a55a78824d-config-volume\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:18.930559 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-cert\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:18.930559 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gzm\" (UniqueName: \"kubernetes.io/projected/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-kube-api-access-l6gzm\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b34d79c2-5815-43a6-aea1-8f328e81c379-crio-socket\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnxq\" (UniqueName: \"kubernetes.io/projected/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-api-access-6cnxq\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519727fa-2299-44e2-a7bd-65a55a78824d-tmp-dir\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnbd\" (UniqueName: \"kubernetes.io/projected/519727fa-2299-44e2-a7bd-65a55a78824d-kube-api-access-glnbd\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b34d79c2-5815-43a6-aea1-8f328e81c379-data-volume\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:18.930807 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:18.930793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.031742 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/519727fa-2299-44e2-a7bd-65a55a78824d-config-volume\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.031852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-cert\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:19.031852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gzm\" (UniqueName: \"kubernetes.io/projected/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-kube-api-access-l6gzm\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:19.031852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b34d79c2-5815-43a6-aea1-8f328e81c379-crio-socket\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.031852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnxq\" (UniqueName: \"kubernetes.io/projected/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-api-access-6cnxq\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.031852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519727fa-2299-44e2-a7bd-65a55a78824d-tmp-dir\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnbd\" (UniqueName: \"kubernetes.io/projected/519727fa-2299-44e2-a7bd-65a55a78824d-kube-api-access-glnbd\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.031920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b34d79c2-5815-43a6-aea1-8f328e81c379-data-volume\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032012    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b34d79c2-5815-43a6-aea1-8f328e81c379-crio-socket\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.032065 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032039    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032078    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.532057145 +0000 UTC m=+34.594396387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs podName:e1632371-d9cd-4766-a6c0-521a31af4fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.032107025 +0000 UTC m=+66.094446268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs") pod "network-metrics-daemon-n9drp" (UID: "e1632371-d9cd-4766-a6c0-521a31af4fe6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519727fa-2299-44e2-a7bd-65a55a78824d-tmp-dir\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032271    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.532317659 +0000 UTC m=+34.594656901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032354    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.032388    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.532379793 +0000 UTC m=+34.594719028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:19.032443 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b34d79c2-5815-43a6-aea1-8f328e81c379-data-volume\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.032751 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.032447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/519727fa-2299-44e2-a7bd-65a55a78824d-config-volume\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.035970 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.035951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-cert\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:19.040286 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.040252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnxq\" (UniqueName: \"kubernetes.io/projected/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-api-access-6cnxq\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.040402 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.040363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnbd\" (UniqueName: \"kubernetes.io/projected/519727fa-2299-44e2-a7bd-65a55a78824d-kube-api-access-glnbd\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.040665 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.040647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gzm\" (UniqueName: \"kubernetes.io/projected/893b03c4-abf1-42e1-ba25-1c1e5e1be64a-kube-api-access-l6gzm\") pod \"ingress-canary-tdz6v\" (UID: \"893b03c4-abf1-42e1-ba25-1c1e5e1be64a\") " pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:19.133388 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.133356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tdz6v"
Sep 16 18:44:19.133619 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.133602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:19.133661 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.133635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:19.133738 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133726    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:19.133769 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133747    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:19.133769 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133756    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sqx8p for pod openshift-network-diagnostics/network-check-target-h7gn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:19.133847 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p podName:46c18fd8-6738-4435-a7dc-6180ef83b6dd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.133782332 +0000 UTC m=+66.196121565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqx8p" (UniqueName: "kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p") pod "network-check-target-h7gn6" (UID: "46c18fd8-6738-4435-a7dc-6180ef83b6dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:19.133847 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133727    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:19.133924 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.133857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret podName:9528a95a-967e-48e3-9caf-5604a16ffa0c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.133841335 +0000 UTC m=+66.196180578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret") pod "global-pull-secret-syncer-xqtnz" (UID: "9528a95a-967e-48e3-9caf-5604a16ffa0c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:19.479525 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.479492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tdz6v"]
Sep 16 18:44:19.483026 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:19.483000    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893b03c4_abf1_42e1_ba25_1c1e5e1be64a.slice/crio-b31735cca1469606c04f72f14f1a36601d7b3fef6eceba48c371856a4c679a20 WatchSource:0}: Error finding container b31735cca1469606c04f72f14f1a36601d7b3fef6eceba48c371856a4c679a20: Status 404 returned error can't find the container with id b31735cca1469606c04f72f14f1a36601d7b3fef6eceba48c371856a4c679a20
Sep 16 18:44:19.536325 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.536273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:19.536325 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.536314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.536452 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.536349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:19.536452 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536413    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:19.536452 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536424    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:19.536543 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536458    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:20.536445531 +0000 UTC m=+35.598784765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:19.536543 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536478    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:20.536465178 +0000 UTC m=+35.598804412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:19.536543 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536424    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:19.536543 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:19.536508    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:20.536497635 +0000 UTC m=+35.598836864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:19.588992 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.588965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tdz6v" event={"ID":"893b03c4-abf1-42e1-ba25-1c1e5e1be64a","Type":"ContainerStarted","Data":"b31735cca1469606c04f72f14f1a36601d7b3fef6eceba48c371856a4c679a20"}
Sep 16 18:44:19.591805 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.591781    2581 generic.go:356] "Generic (PLEG): container finished" podID="b581ff8c-1c56-4015-89bf-b67d85e91e6b" containerID="4ef2821e232ffa6aa17c7aeebe48649c9cbe6a4c01b6fc7b154667c50e836e7a" exitCode=0
Sep 16 18:44:19.591892 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:19.591811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerDied","Data":"4ef2821e232ffa6aa17c7aeebe48649c9cbe6a4c01b6fc7b154667c50e836e7a"}
Sep 16 18:44:20.366407 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.366373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:44:20.387997 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.387954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:44:20.388190 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.388074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.390415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.390388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:44:20.390672 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.390653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:44:20.390774 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.390743    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:44:20.390840 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.390792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wb8gd\""
Sep 16 18:44:20.397204 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.397180    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:44:20.408463 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.408433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:20.408592 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.408475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:20.408592 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.408441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7bds\""
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411168    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:20.411564 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.411483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bb8r2\""
Sep 16 18:44:20.545656 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:20.545656 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.545781    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.545852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.545833962 +0000 UTC m=+37.608173209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.545957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.546000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.546029    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.546071    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.546056717 +0000 UTC m=+37.608395946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.546027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.546101    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.546100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzhc\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.546137    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.546126011 +0000 UTC m=+37.608465243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:20.546210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.546189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.598422 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.598383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tv858" event={"ID":"b581ff8c-1c56-4015-89bf-b67d85e91e6b","Type":"ContainerStarted","Data":"00978385fae002a355243e2107cded83af448f0b30a1a5dea0d4f3ef23b82a6f"}
Sep 16 18:44:20.621210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.621097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tv858" podStartSLOduration=8.061829763 podStartE2EDuration="35.621073971s" totalImagesPullingTime="27.559244208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:20.620707828 +0000 UTC m=+35.683047079" watchObservedRunningTime="2026-09-16 18:44:20.621073971 +0000 UTC m=+35.683413224"
Sep 16 18:44:20.647307 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzhc\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647714 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647714 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.647714 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.647596    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:20.647714 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.647618    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b98bf9c8-9cdsp: secret "image-registry-tls" not found
Sep 16 18:44:20.647714 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:20.647702    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls podName:ff50c590-3f4e-464d-8401-4e513656b4aa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:21.147668584 +0000 UTC m=+36.210007828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls") pod "image-registry-54b98bf9c8-9cdsp" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa") : secret "image-registry-tls" not found
Sep 16 18:44:20.647940 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.647717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.648217 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.648190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.649130 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.649105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.651595 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.651575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.651724 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.651609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.656731 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.656687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:20.657011 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:20.656991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzhc\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:21.152989 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.152950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:21.153199 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:21.153096    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:21.153435 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:21.153406    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b98bf9c8-9cdsp: secret "image-registry-tls" not found
Sep 16 18:44:21.153538 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:21.153523    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls podName:ff50c590-3f4e-464d-8401-4e513656b4aa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.153476873 +0000 UTC m=+37.215816121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls") pod "image-registry-54b98bf9c8-9cdsp" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa") : secret "image-registry-tls" not found
Sep 16 18:44:21.181344 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.181308    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"]
Sep 16 18:44:21.206308 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.206272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"]
Sep 16 18:44:21.206462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.206342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"
Sep 16 18:44:21.209306 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.209274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6cdlv\""
Sep 16 18:44:21.355199 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.355159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmprq\" (UniqueName: \"kubernetes.io/projected/24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2-kube-api-access-kmprq\") pod \"network-check-source-6f8b768455-2hwqk\" (UID: \"24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"
Sep 16 18:44:21.455906 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.455818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmprq\" (UniqueName: \"kubernetes.io/projected/24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2-kube-api-access-kmprq\") pod \"network-check-source-6f8b768455-2hwqk\" (UID: \"24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"
Sep 16 18:44:21.464346 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.464314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmprq\" (UniqueName: \"kubernetes.io/projected/24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2-kube-api-access-kmprq\") pod \"network-check-source-6f8b768455-2hwqk\" (UID: \"24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"
Sep 16 18:44:21.517184 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:21.517130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"
Sep 16 18:44:22.016913 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.016828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk"]
Sep 16 18:44:22.020252 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:22.020209    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bfc5ee_4b9b_460b_95c8_7ea02e5b56f2.slice/crio-2aa9d240733fb01330b155662cce22d41c33581a8b4382e148de5eeabf9b622d WatchSource:0}: Error finding container 2aa9d240733fb01330b155662cce22d41c33581a8b4382e148de5eeabf9b622d: Status 404 returned error can't find the container with id 2aa9d240733fb01330b155662cce22d41c33581a8b4382e148de5eeabf9b622d
Sep 16 18:44:22.162491 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.162460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:22.162683 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.162611    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:22.162683 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.162631    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b98bf9c8-9cdsp: secret "image-registry-tls" not found
Sep 16 18:44:22.162756 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.162690    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls podName:ff50c590-3f4e-464d-8401-4e513656b4aa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:24.162671271 +0000 UTC m=+39.225010499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls") pod "image-registry-54b98bf9c8-9cdsp" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa") : secret "image-registry-tls" not found
Sep 16 18:44:22.200750 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.200716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"]
Sep 16 18:44:22.215374 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.215344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"]
Sep 16 18:44:22.215523 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.215448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.217863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.217833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:22.218168 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.218129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-27rzk\""
Sep 16 18:44:22.218637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.218617    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:44:22.218637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.218631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:22.218824 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.218670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:44:22.364535 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.364503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc51e700-c58d-48ce-b099-e448440ebf32-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.364535 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.364540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv5p\" (UniqueName: \"kubernetes.io/projected/cc51e700-c58d-48ce-b099-e448440ebf32-kube-api-access-7hv5p\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.364741 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.364559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc51e700-c58d-48ce-b099-e448440ebf32-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.465350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.465309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc51e700-c58d-48ce-b099-e448440ebf32-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.465350 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.465346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv5p\" (UniqueName: \"kubernetes.io/projected/cc51e700-c58d-48ce-b099-e448440ebf32-kube-api-access-7hv5p\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.465548 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.465366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc51e700-c58d-48ce-b099-e448440ebf32-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.465925 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.465905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc51e700-c58d-48ce-b099-e448440ebf32-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.468487 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.468460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc51e700-c58d-48ce-b099-e448440ebf32-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.473652 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.473628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv5p\" (UniqueName: \"kubernetes.io/projected/cc51e700-c58d-48ce-b099-e448440ebf32-kube-api-access-7hv5p\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lhw7j\" (UID: \"cc51e700-c58d-48ce-b099-e448440ebf32\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.525993 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.525955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"
Sep 16 18:44:22.565795 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.565757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:22.565981 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.565823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:22.565981 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.565950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:22.566126 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566110    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:22.566219 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.566184395 +0000 UTC m=+41.628523624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:22.566636 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566612    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:22.566795 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566671    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.566654711 +0000 UTC m=+41.628993956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:22.566795 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566746    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:22.566795 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:22.566782    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.56677062 +0000 UTC m=+41.629109853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:22.606627 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.606591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tdz6v" event={"ID":"893b03c4-abf1-42e1-ba25-1c1e5e1be64a","Type":"ContainerStarted","Data":"af41718f38a8c4f680bfc5d3b4076a3868da4288c70ffdeb2bf3fed074b2bc9f"}
Sep 16 18:44:22.610383 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.610297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk" event={"ID":"24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2","Type":"ContainerStarted","Data":"2aa9d240733fb01330b155662cce22d41c33581a8b4382e148de5eeabf9b622d"}
Sep 16 18:44:22.625608 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.625504    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tdz6v" podStartSLOduration=2.070327472 podStartE2EDuration="4.625486916s" totalImagesPullingTime="2.555159444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:22.62520481 +0000 UTC m=+37.687544062" watchObservedRunningTime="2026-09-16 18:44:22.625486916 +0000 UTC m=+37.687826167"
Sep 16 18:44:22.819022 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:22.818982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j"]
Sep 16 18:44:22.822463 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:22.822431    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc51e700_c58d_48ce_b099_e448440ebf32.slice/crio-3d71d154339aadc5af16338514c59720915b81825e397915976470b55a5f9501 WatchSource:0}: Error finding container 3d71d154339aadc5af16338514c59720915b81825e397915976470b55a5f9501: Status 404 returned error can't find the container with id 3d71d154339aadc5af16338514c59720915b81825e397915976470b55a5f9501
Sep 16 18:44:23.614970 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:23.614928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j" event={"ID":"cc51e700-c58d-48ce-b099-e448440ebf32","Type":"ContainerStarted","Data":"3d71d154339aadc5af16338514c59720915b81825e397915976470b55a5f9501"}
Sep 16 18:44:24.180897 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:24.180860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:24.181136 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:24.181052    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:24.181136 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:24.181073    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b98bf9c8-9cdsp: secret "image-registry-tls" not found
Sep 16 18:44:24.181326 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:24.181166    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls podName:ff50c590-3f4e-464d-8401-4e513656b4aa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:28.181130598 +0000 UTC m=+43.243469827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls") pod "image-registry-54b98bf9c8-9cdsp" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa") : secret "image-registry-tls" not found
Sep 16 18:44:26.600438 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.600399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.600453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.600484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600578    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600581    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600604    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600633    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.600619416 +0000 UTC m=+49.662958644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600646    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.600640108 +0000 UTC m=+49.662979337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:26.600861 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:26.600656    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.600650857 +0000 UTC m=+49.662990085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:26.624314 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.624278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk" event={"ID":"24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2","Type":"ContainerStarted","Data":"de72f8a514cab8fb704cc55c8a0f4d63276dcd7736b9dc363d3109789e8e1478"}
Sep 16 18:44:26.625982 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.625946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j" event={"ID":"cc51e700-c58d-48ce-b099-e448440ebf32","Type":"ContainerStarted","Data":"ffa220c3b7b6c1040c860afff0afdc45c8d5bea778883e0c5ef800a5c6407c9c"}
Sep 16 18:44:26.655196 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.655120    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2hwqk" podStartSLOduration=1.7697455290000002 podStartE2EDuration="5.655099374s" totalImagesPullingTime="3.885353845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:26.651127544 +0000 UTC m=+41.713466798" watchObservedRunningTime="2026-09-16 18:44:26.655099374 +0000 UTC m=+41.717438628"
Sep 16 18:44:26.682810 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:26.682757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j" podStartSLOduration=1.584080141 podStartE2EDuration="4.682739849s" totalImagesPullingTime="3.098659708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:26.682562898 +0000 UTC m=+41.744902149" watchObservedRunningTime="2026-09-16 18:44:26.682739849 +0000 UTC m=+41.745079091"
Sep 16 18:44:28.214815 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.214778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:28.215215 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:28.214939    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:28.215215 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:28.214963    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b98bf9c8-9cdsp: secret "image-registry-tls" not found
Sep 16 18:44:28.215215 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:28.215024    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls podName:ff50c590-3f4e-464d-8401-4e513656b4aa nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.215001477 +0000 UTC m=+51.277340706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls") pod "image-registry-54b98bf9c8-9cdsp" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa") : secret "image-registry-tls" not found
Sep 16 18:44:28.383472 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.383437    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"]
Sep 16 18:44:28.403489 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.403449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"]
Sep 16 18:44:28.403683 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.403590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"
Sep 16 18:44:28.406452 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.406427    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:28.406669 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.406650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gfcvz\""
Sep 16 18:44:28.407605 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.407581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:44:28.516895 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.516783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j64s\" (UniqueName: \"kubernetes.io/projected/b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79-kube-api-access-2j64s\") pod \"migrator-8b4d697c5-p7lhc\" (UID: \"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"
Sep 16 18:44:28.617664 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.617623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j64s\" (UniqueName: \"kubernetes.io/projected/b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79-kube-api-access-2j64s\") pod \"migrator-8b4d697c5-p7lhc\" (UID: \"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"
Sep 16 18:44:28.626303 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.626274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j64s\" (UniqueName: \"kubernetes.io/projected/b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79-kube-api-access-2j64s\") pod \"migrator-8b4d697c5-p7lhc\" (UID: \"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"
Sep 16 18:44:28.713922 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:28.713885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"
Sep 16 18:44:29.013258 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:29.013224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc"]
Sep 16 18:44:29.016473 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:29.016448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a6102f_c3e9_48ae_9a17_9dd17a5f3c79.slice/crio-8cfc030e5cff32eb515a7c60fe17a442aed738798c061c164e56a606d006ff1a WatchSource:0}: Error finding container 8cfc030e5cff32eb515a7c60fe17a442aed738798c061c164e56a606d006ff1a: Status 404 returned error can't find the container with id 8cfc030e5cff32eb515a7c60fe17a442aed738798c061c164e56a606d006ff1a
Sep 16 18:44:29.635445 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:29.635407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc" event={"ID":"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79","Type":"ContainerStarted","Data":"8cfc030e5cff32eb515a7c60fe17a442aed738798c061c164e56a606d006ff1a"}
Sep 16 18:44:31.641684 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:31.641646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc" event={"ID":"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79","Type":"ContainerStarted","Data":"08363de41c62ee4b3dc48697d932f7b3e15c7fd32bdcaaa1baeb8628fe0a65ef"}
Sep 16 18:44:31.642059 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:31.641705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc" event={"ID":"b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79","Type":"ContainerStarted","Data":"c89298c78a365c81183783e5f5f98ed11d48804a80c91762c93f691ef61dfd85"}
Sep 16 18:44:31.669184 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:31.669117    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p7lhc" podStartSLOduration=2.12414327 podStartE2EDuration="3.669101692s" totalImagesPullingTime="1.544958422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:31.667640998 +0000 UTC m=+46.729980249" watchObservedRunningTime="2026-09-16 18:44:31.669101692 +0000 UTC m=+46.731440942"
Sep 16 18:44:34.666123 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:34.666086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:34.666137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:34.666290    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:34.666290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:34.666353    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:34.666366    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls podName:519727fa-2299-44e2-a7bd-65a55a78824d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:50.666344953 +0000 UTC m=+65.728684183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls") pod "dns-default-6pt88" (UID: "519727fa-2299-44e2-a7bd-65a55a78824d") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:44:34.666422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls podName:b34d79c2-5815-43a6-aea1-8f328e81c379 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:50.666400478 +0000 UTC m=+65.728739722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s55fk" (UID: "b34d79c2-5815-43a6-aea1-8f328e81c379") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:34.666699 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:34.666659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b34d79c2-5815-43a6-aea1-8f328e81c379-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:36.279614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.279569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:36.281956 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.281924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"image-registry-54b98bf9c8-9cdsp\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") " pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:36.299843 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.299816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:36.569669 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.569587    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:44:36.573133 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:36.573099    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff50c590_3f4e_464d_8401_4e513656b4aa.slice/crio-d645f49bb8e8419e30f21c496e78aeb8231964df3a943ec0ecc83b353d3c475d WatchSource:0}: Error finding container d645f49bb8e8419e30f21c496e78aeb8231964df3a943ec0ecc83b353d3c475d: Status 404 returned error can't find the container with id d645f49bb8e8419e30f21c496e78aeb8231964df3a943ec0ecc83b353d3c475d
Sep 16 18:44:36.657355 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.657314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" event={"ID":"ff50c590-3f4e-464d-8401-4e513656b4aa","Type":"ContainerStarted","Data":"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"}
Sep 16 18:44:36.657495 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.657375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" event={"ID":"ff50c590-3f4e-464d-8401-4e513656b4aa","Type":"ContainerStarted","Data":"d645f49bb8e8419e30f21c496e78aeb8231964df3a943ec0ecc83b353d3c475d"}
Sep 16 18:44:36.657495 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.657467    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:44:36.677859 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:36.677803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" podStartSLOduration=16.67778484 podStartE2EDuration="16.67778484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:36.676917276 +0000 UTC m=+51.739256540" watchObservedRunningTime="2026-09-16 18:44:36.67778484 +0000 UTC m=+51.740124099"
Sep 16 18:44:44.582072 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:44.582042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knw5n"
Sep 16 18:44:50.212778 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.212738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"]
Sep 16 18:44:50.216296 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.216275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.218614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.218583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:44:50.218744 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.218649    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:44:50.219748 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.219725    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:44:50.219845 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.219759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:44:50.237135 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.237103    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"]
Sep 16 18:44:50.283546 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.283508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s558k\" (UniqueName: \"kubernetes.io/projected/e444c8ea-99e7-4ef8-904b-b79359878ccb-kube-api-access-s558k\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.283546 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.283551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e444c8ea-99e7-4ef8-904b-b79359878ccb-tmp\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.283764 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.283577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e444c8ea-99e7-4ef8-904b-b79359878ccb-klusterlet-config\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.292069 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.292038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:44:50.351133 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.351098    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"]
Sep 16 18:44:50.353074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.353059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:50.365002 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.364982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t2zxc\""
Sep 16 18:44:50.365260 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.365245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:44:50.377459 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.377429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f9bd6d57-bggql"]
Sep 16 18:44:50.379364 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.379345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.384550 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.384522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b79d70dc-5358-4de3-b19a-0fab21815a56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jzt7z\" (UID: \"b79d70dc-5358-4de3-b19a-0fab21815a56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:50.384663 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.384575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s558k\" (UniqueName: \"kubernetes.io/projected/e444c8ea-99e7-4ef8-904b-b79359878ccb-kube-api-access-s558k\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.384663 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.384614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e444c8ea-99e7-4ef8-904b-b79359878ccb-tmp\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.384663 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.384642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e444c8ea-99e7-4ef8-904b-b79359878ccb-klusterlet-config\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.385103 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.385079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e444c8ea-99e7-4ef8-904b-b79359878ccb-tmp\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.387135 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.387113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e444c8ea-99e7-4ef8-904b-b79359878ccb-klusterlet-config\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.398108 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.398084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"]
Sep 16 18:44:50.403773 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.403749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9bd6d57-bggql"]
Sep 16 18:44:50.407339 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.407313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s558k\" (UniqueName: \"kubernetes.io/projected/e444c8ea-99e7-4ef8-904b-b79359878ccb-kube-api-access-s558k\") pod \"klusterlet-addon-workmgr-84445df79d-gk4xj\" (UID: \"e444c8ea-99e7-4ef8-904b-b79359878ccb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.485779 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-certificates\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485779 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-trusted-ca\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485779 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-image-registry-private-configuration\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-installation-pull-secrets\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-tls\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-bound-sa-token\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b79d70dc-5358-4de3-b19a-0fab21815a56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jzt7z\" (UID: \"b79d70dc-5358-4de3-b19a-0fab21815a56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxfr\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-kube-api-access-vsxfr\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.485998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.485997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f62a0da4-c4e6-4c89-a9d7-6a3654803849-ca-trust-extracted\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.488408 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.488384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b79d70dc-5358-4de3-b19a-0fab21815a56-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jzt7z\" (UID: \"b79d70dc-5358-4de3-b19a-0fab21815a56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:50.525280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.525253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:50.587181 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-tls\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587315 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-bound-sa-token\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587315 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxfr\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-kube-api-access-vsxfr\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587315 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f62a0da4-c4e6-4c89-a9d7-6a3654803849-ca-trust-extracted\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587414 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-certificates\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587414 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-trusted-ca\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587474 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-image-registry-private-configuration\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587474 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-installation-pull-secrets\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.587852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.587813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f62a0da4-c4e6-4c89-a9d7-6a3654803849-ca-trust-extracted\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.588335 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.588310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-certificates\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.588719 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.588694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f62a0da4-c4e6-4c89-a9d7-6a3654803849-trusted-ca\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.589729 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.589706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-image-registry-private-configuration\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.589972 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.589912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-registry-tls\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.590352 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.590327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f62a0da4-c4e6-4c89-a9d7-6a3654803849-installation-pull-secrets\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.596186 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.596144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-bound-sa-token\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.596310 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.596294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxfr\" (UniqueName: \"kubernetes.io/projected/f62a0da4-c4e6-4c89-a9d7-6a3654803849-kube-api-access-vsxfr\") pod \"image-registry-f9bd6d57-bggql\" (UID: \"f62a0da4-c4e6-4c89-a9d7-6a3654803849\") " pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.661933 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.661884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:50.687748 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.687717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:50.687919 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.687890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:50.688026 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.688010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:50.690339 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.690316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/519727fa-2299-44e2-a7bd-65a55a78824d-metrics-tls\") pod \"dns-default-6pt88\" (UID: \"519727fa-2299-44e2-a7bd-65a55a78824d\") " pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:50.690497 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.690346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b34d79c2-5815-43a6-aea1-8f328e81c379-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s55fk\" (UID: \"b34d79c2-5815-43a6-aea1-8f328e81c379\") " pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:50.797308 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.797278    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"]
Sep 16 18:44:50.800026 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:50.799997    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode444c8ea_99e7_4ef8_904b_b79359878ccb.slice/crio-6ad9512e653cf75316b2dea67764f8ef26f13918196f392a8d1f81fad3605b65 WatchSource:0}: Error finding container 6ad9512e653cf75316b2dea67764f8ef26f13918196f392a8d1f81fad3605b65: Status 404 returned error can't find the container with id 6ad9512e653cf75316b2dea67764f8ef26f13918196f392a8d1f81fad3605b65
Sep 16 18:44:50.930355 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.930327    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"]
Sep 16 18:44:50.932714 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:50.932690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79d70dc_5358_4de3_b19a_0fab21815a56.slice/crio-988ce82fe2d524926d7a2e1e86e6eb7076d3146945a3965bab05169310dec700 WatchSource:0}: Error finding container 988ce82fe2d524926d7a2e1e86e6eb7076d3146945a3965bab05169310dec700: Status 404 returned error can't find the container with id 988ce82fe2d524926d7a2e1e86e6eb7076d3146945a3965bab05169310dec700
Sep 16 18:44:50.941292 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.941269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlsxc\""
Sep 16 18:44:50.945709 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.945690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6mpm2\""
Sep 16 18:44:50.949460 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.949442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s55fk"
Sep 16 18:44:50.954808 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.954787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:50.959660 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:50.959638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f9bd6d57-bggql"]
Sep 16 18:44:50.962867 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:50.962839    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62a0da4_c4e6_4c89_a9d7_6a3654803849.slice/crio-ef4a2221adc3696a5246cbadb22e87bd66db10c87378a0c8439f293bee890686 WatchSource:0}: Error finding container ef4a2221adc3696a5246cbadb22e87bd66db10c87378a0c8439f293bee890686: Status 404 returned error can't find the container with id ef4a2221adc3696a5246cbadb22e87bd66db10c87378a0c8439f293bee890686
Sep 16 18:44:51.091486 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.091446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:51.093804 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.093778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:51.104713 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.104684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1632371-d9cd-4766-a6c0-521a31af4fe6-metrics-certs\") pod \"network-metrics-daemon-n9drp\" (UID: \"e1632371-d9cd-4766-a6c0-521a31af4fe6\") " pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:51.191987 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.191950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:51.191987 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.191989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:51.194480 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.194455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:51.194591 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.194487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx8p\" (UniqueName: \"kubernetes.io/projected/46c18fd8-6738-4435-a7dc-6180ef83b6dd-kube-api-access-sqx8p\") pod \"network-check-target-h7gn6\" (UID: \"46c18fd8-6738-4435-a7dc-6180ef83b6dd\") " pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:51.204673 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.204646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9528a95a-967e-48e3-9caf-5604a16ffa0c-original-pull-secret\") pod \"global-pull-secret-syncer-xqtnz\" (UID: \"9528a95a-967e-48e3-9caf-5604a16ffa0c\") " pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:51.220801 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.220767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s55fk"]
Sep 16 18:44:51.223970 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:51.223925    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34d79c2_5815_43a6_aea1_8f328e81c379.slice/crio-034bbdafb6ee1e2de40b5adce6c3d86517e2d4b13db41b7cd5fa998fff7edd6f WatchSource:0}: Error finding container 034bbdafb6ee1e2de40b5adce6c3d86517e2d4b13db41b7cd5fa998fff7edd6f: Status 404 returned error can't find the container with id 034bbdafb6ee1e2de40b5adce6c3d86517e2d4b13db41b7cd5fa998fff7edd6f
Sep 16 18:44:51.232434 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.232411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pt88"]
Sep 16 18:44:51.235655 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:51.235621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519727fa_2299_44e2_a7bd_65a55a78824d.slice/crio-dd47d6f7d45ee6d3aa6d94fec0c87f63d56f5b23b034efaff7c1edb7eca13eb4 WatchSource:0}: Error finding container dd47d6f7d45ee6d3aa6d94fec0c87f63d56f5b23b034efaff7c1edb7eca13eb4: Status 404 returned error can't find the container with id dd47d6f7d45ee6d3aa6d94fec0c87f63d56f5b23b034efaff7c1edb7eca13eb4
Sep 16 18:44:51.322089 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.322057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7bds\""
Sep 16 18:44:51.328957 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.328927    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bb8r2\""
Sep 16 18:44:51.329962 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.329935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9drp"
Sep 16 18:44:51.331546 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.331529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqtnz"
Sep 16 18:44:51.337133 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.337115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:51.605475 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.605444    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9drp"]
Sep 16 18:44:51.608362 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.608338    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqtnz"]
Sep 16 18:44:51.609495 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:51.609471    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1632371_d9cd_4766_a6c0_521a31af4fe6.slice/crio-a7183eeda4144ee40491b8741d543a0f39f20216267e53e62c75f468001159c6 WatchSource:0}: Error finding container a7183eeda4144ee40491b8741d543a0f39f20216267e53e62c75f468001159c6: Status 404 returned error can't find the container with id a7183eeda4144ee40491b8741d543a0f39f20216267e53e62c75f468001159c6
Sep 16 18:44:51.611931 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:51.611906    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9528a95a_967e_48e3_9caf_5604a16ffa0c.slice/crio-34faeae591b68b2200b968d2ad3f75aaa2ebc0510d0aa71d8ac152404341af27 WatchSource:0}: Error finding container 34faeae591b68b2200b968d2ad3f75aaa2ebc0510d0aa71d8ac152404341af27: Status 404 returned error can't find the container with id 34faeae591b68b2200b968d2ad3f75aaa2ebc0510d0aa71d8ac152404341af27
Sep 16 18:44:51.635608 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.635575    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7gn6"]
Sep 16 18:44:51.640106 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:51.640074    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c18fd8_6738_4435_a7dc_6180ef83b6dd.slice/crio-7711b3abee656be09088ba4dc708713e2914532b706431e4624fa6a216ec0e91 WatchSource:0}: Error finding container 7711b3abee656be09088ba4dc708713e2914532b706431e4624fa6a216ec0e91: Status 404 returned error can't find the container with id 7711b3abee656be09088ba4dc708713e2914532b706431e4624fa6a216ec0e91
Sep 16 18:44:51.716183 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.716106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqtnz" event={"ID":"9528a95a-967e-48e3-9caf-5604a16ffa0c","Type":"ContainerStarted","Data":"34faeae591b68b2200b968d2ad3f75aaa2ebc0510d0aa71d8ac152404341af27"}
Sep 16 18:44:51.718649 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.718611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9drp" event={"ID":"e1632371-d9cd-4766-a6c0-521a31af4fe6","Type":"ContainerStarted","Data":"a7183eeda4144ee40491b8741d543a0f39f20216267e53e62c75f468001159c6"}
Sep 16 18:44:51.721035 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.720982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s55fk" event={"ID":"b34d79c2-5815-43a6-aea1-8f328e81c379","Type":"ContainerStarted","Data":"2b22ee1a757b930008037ba761c4a434d459eb2397c5a39c559ebdc001518ec7"}
Sep 16 18:44:51.721195 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.721044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s55fk" event={"ID":"b34d79c2-5815-43a6-aea1-8f328e81c379","Type":"ContainerStarted","Data":"034bbdafb6ee1e2de40b5adce6c3d86517e2d4b13db41b7cd5fa998fff7edd6f"}
Sep 16 18:44:51.723535 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.723510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7gn6" event={"ID":"46c18fd8-6738-4435-a7dc-6180ef83b6dd","Type":"ContainerStarted","Data":"5b6a209283b4086331b8341ad31005d62abb607a506328c53312243215ab6aa1"}
Sep 16 18:44:51.723643 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.723552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7gn6" event={"ID":"46c18fd8-6738-4435-a7dc-6180ef83b6dd","Type":"ContainerStarted","Data":"7711b3abee656be09088ba4dc708713e2914532b706431e4624fa6a216ec0e91"}
Sep 16 18:44:51.723781 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.723717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:44:51.726926 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.726883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9bd6d57-bggql" event={"ID":"f62a0da4-c4e6-4c89-a9d7-6a3654803849","Type":"ContainerStarted","Data":"6709da75538b0c442b78d700cad3f8548390f196f4790947c2555282e4dd3727"}
Sep 16 18:44:51.726926 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.726924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f9bd6d57-bggql" event={"ID":"f62a0da4-c4e6-4c89-a9d7-6a3654803849","Type":"ContainerStarted","Data":"ef4a2221adc3696a5246cbadb22e87bd66db10c87378a0c8439f293bee890686"}
Sep 16 18:44:51.728362 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.727352    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:44:51.728469 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.728431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pt88" event={"ID":"519727fa-2299-44e2-a7bd-65a55a78824d","Type":"ContainerStarted","Data":"dd47d6f7d45ee6d3aa6d94fec0c87f63d56f5b23b034efaff7c1edb7eca13eb4"}
Sep 16 18:44:51.730137 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.730035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z" event={"ID":"b79d70dc-5358-4de3-b19a-0fab21815a56","Type":"ContainerStarted","Data":"988ce82fe2d524926d7a2e1e86e6eb7076d3146945a3965bab05169310dec700"}
Sep 16 18:44:51.733115 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.733055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj" event={"ID":"e444c8ea-99e7-4ef8-904b-b79359878ccb","Type":"ContainerStarted","Data":"6ad9512e653cf75316b2dea67764f8ef26f13918196f392a8d1f81fad3605b65"}
Sep 16 18:44:51.765043 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:51.763533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h7gn6" podStartSLOduration=66.763513105 podStartE2EDuration="1m6.763513105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.74377264 +0000 UTC m=+66.806111892" watchObservedRunningTime="2026-09-16 18:44:51.763513105 +0000 UTC m=+66.825852357"
Sep 16 18:44:52.742067 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.741880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s55fk" event={"ID":"b34d79c2-5815-43a6-aea1-8f328e81c379","Type":"ContainerStarted","Data":"0e3dbd8b514ac4d609b3adf4058c0daf161da5108bc16b01dec77ee6b92afc0c"}
Sep 16 18:44:52.746199 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.745203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z" event={"ID":"b79d70dc-5358-4de3-b19a-0fab21815a56","Type":"ContainerStarted","Data":"3894cc912ef5dbc0baed26dce220ef9bac6b10fa340de4796b82b552add849c4"}
Sep 16 18:44:52.746199 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.746185    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:52.752894 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.752828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z"
Sep 16 18:44:52.771419 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.770747    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jzt7z" podStartSLOduration=1.44294773 podStartE2EDuration="2.770723858s" totalImagesPullingTime="1.327776128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.769252624 +0000 UTC m=+67.831591877" watchObservedRunningTime="2026-09-16 18:44:52.770723858 +0000 UTC m=+67.833063110"
Sep 16 18:44:52.771419 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:52.771010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f9bd6d57-bggql" podStartSLOduration=2.770999742 podStartE2EDuration="2.770999742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.765024176 +0000 UTC m=+66.827363428" watchObservedRunningTime="2026-09-16 18:44:52.770999742 +0000 UTC m=+67.833338988"
Sep 16 18:44:53.045519 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.045278    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"]
Sep 16 18:44:53.047375 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.047350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.049794 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.049770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:44:53.050771 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.050747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.050870 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.050795    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.051597 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.050991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:44:53.051597 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.051202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r9v74\""
Sep 16 18:44:53.051597 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.051485    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:44:53.058491 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.058466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"]
Sep 16 18:44:53.109512 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.109472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbvj\" (UniqueName: \"kubernetes.io/projected/0f61d792-9cc5-405f-91e4-b4a66878e63a-kube-api-access-fjbvj\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.109679 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.109573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.109679 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.109617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.109679 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.109651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f61d792-9cc5-405f-91e4-b4a66878e63a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.210698 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.210656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbvj\" (UniqueName: \"kubernetes.io/projected/0f61d792-9cc5-405f-91e4-b4a66878e63a-kube-api-access-fjbvj\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.210958 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.210737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.210958 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.210779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.210958 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.210808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f61d792-9cc5-405f-91e4-b4a66878e63a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.211627 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.211578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f61d792-9cc5-405f-91e4-b4a66878e63a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.214428 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.214375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.214646 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.214602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f61d792-9cc5-405f-91e4-b4a66878e63a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.225572 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.225518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbvj\" (UniqueName: \"kubernetes.io/projected/0f61d792-9cc5-405f-91e4-b4a66878e63a-kube-api-access-fjbvj\") pod \"prometheus-operator-68bcd7f987-km4cp\" (UID: \"0f61d792-9cc5-405f-91e4-b4a66878e63a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:53.361884 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:53.361735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"
Sep 16 18:44:58.454920 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.454846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km4cp"]
Sep 16 18:44:58.461321 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:44:58.461288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f61d792_9cc5_405f_91e4_b4a66878e63a.slice/crio-44881e6a5a9e2dd1ee241d5854df4f47035f6b8e8a7ad570d820b1d7325735ff WatchSource:0}: Error finding container 44881e6a5a9e2dd1ee241d5854df4f47035f6b8e8a7ad570d820b1d7325735ff: Status 404 returned error can't find the container with id 44881e6a5a9e2dd1ee241d5854df4f47035f6b8e8a7ad570d820b1d7325735ff
Sep 16 18:44:58.777232 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.777121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj" event={"ID":"e444c8ea-99e7-4ef8-904b-b79359878ccb","Type":"ContainerStarted","Data":"801194e55150fc0938c0d2da1003ea8cdfb751d48f66e67cd02467933d5a3f3b"}
Sep 16 18:44:58.777385 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.777330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:58.778942 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.778910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqtnz" event={"ID":"9528a95a-967e-48e3-9caf-5604a16ffa0c","Type":"ContainerStarted","Data":"44a74321e5e137b27a6e687d882eda5b299fdaae56d9de98ce7d4431ede5a79f"}
Sep 16 18:44:58.779239 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.779218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj"
Sep 16 18:44:58.780644 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.780626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9drp" event={"ID":"e1632371-d9cd-4766-a6c0-521a31af4fe6","Type":"ContainerStarted","Data":"69a3a873966de817828080fa5857a5090e852e348b554c58bc772f0dd061aba2"}
Sep 16 18:44:58.780728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.780661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9drp" event={"ID":"e1632371-d9cd-4766-a6c0-521a31af4fe6","Type":"ContainerStarted","Data":"a7a3f685cdc4bdaa47525f0565ae30ca8cbf8dace905c9bfbc1df0c81e6e2ee3"}
Sep 16 18:44:58.782671 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.782649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s55fk" event={"ID":"b34d79c2-5815-43a6-aea1-8f328e81c379","Type":"ContainerStarted","Data":"9680ff2e9da2947c27a82662799ff6221a1d718a35845e7aff43a4589144f7e8"}
Sep 16 18:44:58.783775 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.783752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp" event={"ID":"0f61d792-9cc5-405f-91e4-b4a66878e63a","Type":"ContainerStarted","Data":"44881e6a5a9e2dd1ee241d5854df4f47035f6b8e8a7ad570d820b1d7325735ff"}
Sep 16 18:44:58.785383 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.785363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pt88" event={"ID":"519727fa-2299-44e2-a7bd-65a55a78824d","Type":"ContainerStarted","Data":"1b6e20922a60aba26e474ea291861ab65c6b1a6d1b3bbd2d0b942117c8b36f1b"}
Sep 16 18:44:58.785469 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.785396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pt88" event={"ID":"519727fa-2299-44e2-a7bd-65a55a78824d","Type":"ContainerStarted","Data":"3e5ac599218818580f5569ab980458a9c9b29ad4edfc182e494b20f03223907b"}
Sep 16 18:44:58.785511 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.785505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6pt88"
Sep 16 18:44:58.796647 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.796603    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84445df79d-gk4xj" podStartSLOduration=1.408659998 podStartE2EDuration="8.796589617s" totalImagesPullingTime="7.387929619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.795357743 +0000 UTC m=+73.857696994" watchObservedRunningTime="2026-09-16 18:44:58.796589617 +0000 UTC m=+73.858928867"
Sep 16 18:44:58.831200 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.831132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n9drp" podStartSLOduration=67.254363019 podStartE2EDuration="1m13.831117311s" totalImagesPullingTime="6.576754292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.830261559 +0000 UTC m=+73.892600809" watchObservedRunningTime="2026-09-16 18:44:58.831117311 +0000 UTC m=+73.893456558"
Sep 16 18:44:58.848347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.848297    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xqtnz" podStartSLOduration=67.271075439 podStartE2EDuration="1m13.848281503s" totalImagesPullingTime="6.577206064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.84758444 +0000 UTC m=+73.909923692" watchObservedRunningTime="2026-09-16 18:44:58.848281503 +0000 UTC m=+73.910620751"
Sep 16 18:44:58.866899 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.866851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s55fk" podStartSLOduration=33.800181187 podStartE2EDuration="40.866837631s" totalImagesPullingTime="7.066656444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.865955796 +0000 UTC m=+73.928295047" watchObservedRunningTime="2026-09-16 18:44:58.866837631 +0000 UTC m=+73.929176881"
Sep 16 18:44:58.884821 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:58.884770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6pt88" podStartSLOduration=33.936266641 podStartE2EDuration="40.884755816s" totalImagesPullingTime="6.948489175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.883337695 +0000 UTC m=+73.945676946" watchObservedRunningTime="2026-09-16 18:44:58.884755816 +0000 UTC m=+73.947095066"
Sep 16 18:44:59.793756 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:59.793712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp" event={"ID":"0f61d792-9cc5-405f-91e4-b4a66878e63a","Type":"ContainerStarted","Data":"580cf34fcd5da0a409890f0124c38f656aa0b95943e03814a9f5abc86bfc2300"}
Sep 16 18:44:59.794218 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:59.793789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp" event={"ID":"0f61d792-9cc5-405f-91e4-b4a66878e63a","Type":"ContainerStarted","Data":"42cf405cb241925bd81b4e053cf559bb4bb4a75364975d5a876f31973fcc56aa"}
Sep 16 18:44:59.824856 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:44:59.824754    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km4cp" podStartSLOduration=5.734299345 podStartE2EDuration="6.824735596s" totalImagesPullingTime="1.090436251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:59.823739727 +0000 UTC m=+74.886078980" watchObservedRunningTime="2026-09-16 18:44:59.824735596 +0000 UTC m=+74.887074846"
Sep 16 18:45:00.297286 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:00.297247    2581 patch_prober.go:28] interesting pod/image-registry-54b98bf9c8-9cdsp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:45:00.297451 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:00.297301    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:45:01.451020 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.450988    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fxcdf"]
Sep 16 18:45:01.453400 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.453377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.457800 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.457773    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:45:01.457911 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.457851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:45:01.457911 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.457851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k6sj2\""
Sep 16 18:45:01.458028 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.457894    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:45:01.572796 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-metrics-client-ca\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.572796 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-root\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-tls\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-wtmp\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.572949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lfd\" (UniqueName: \"kubernetes.io/projected/08ab8584-5ac2-4ab2-9bf9-007c943a0750-kube-api-access-66lfd\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.573007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-sys\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.573032 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.573028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-textfile\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674276 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-sys\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674276 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-textfile\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-metrics-client-ca\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-root\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-sys\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-tls\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-wtmp\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-root\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lfd\" (UniqueName: \"kubernetes.io/projected/08ab8584-5ac2-4ab2-9bf9-007c943a0750-kube-api-access-66lfd\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.674611 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-wtmp\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.675015 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-textfile\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.675015 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-metrics-client-ca\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.675015 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.674957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.676858 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.676830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-tls\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.676970 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.676830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08ab8584-5ac2-4ab2-9bf9-007c943a0750-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.682376 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.682349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lfd\" (UniqueName: \"kubernetes.io/projected/08ab8584-5ac2-4ab2-9bf9-007c943a0750-kube-api-access-66lfd\") pod \"node-exporter-fxcdf\" (UID: \"08ab8584-5ac2-4ab2-9bf9-007c943a0750\") " pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.762367 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.762290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fxcdf"
Sep 16 18:45:01.772035 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:45:01.772009    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ab8584_5ac2_4ab2_9bf9_007c943a0750.slice/crio-bb3e636598231518e8ac0897ae942595a8f6e966a211e7b303fb60ca90d8b615 WatchSource:0}: Error finding container bb3e636598231518e8ac0897ae942595a8f6e966a211e7b303fb60ca90d8b615: Status 404 returned error can't find the container with id bb3e636598231518e8ac0897ae942595a8f6e966a211e7b303fb60ca90d8b615
Sep 16 18:45:01.804786 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:01.804748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxcdf" event={"ID":"08ab8584-5ac2-4ab2-9bf9-007c943a0750","Type":"ContainerStarted","Data":"bb3e636598231518e8ac0897ae942595a8f6e966a211e7b303fb60ca90d8b615"}
Sep 16 18:45:02.809858 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:02.809817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxcdf" event={"ID":"08ab8584-5ac2-4ab2-9bf9-007c943a0750","Type":"ContainerStarted","Data":"8a63d92ebccb7ea024b2e82b49d2c0d0997d3f7a0a173373f9644c1aaa778db9"}
Sep 16 18:45:03.816941 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:03.816900    2581 generic.go:356] "Generic (PLEG): container finished" podID="08ab8584-5ac2-4ab2-9bf9-007c943a0750" containerID="8a63d92ebccb7ea024b2e82b49d2c0d0997d3f7a0a173373f9644c1aaa778db9" exitCode=0
Sep 16 18:45:03.817362 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:03.816960    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxcdf" event={"ID":"08ab8584-5ac2-4ab2-9bf9-007c943a0750","Type":"ContainerDied","Data":"8a63d92ebccb7ea024b2e82b49d2c0d0997d3f7a0a173373f9644c1aaa778db9"}
Sep 16 18:45:04.414702 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.414665    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54757d56d8-k6qw8"]
Sep 16 18:45:04.417224 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.417208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.419512 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:45:04.419642 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bsmrv\""
Sep 16 18:45:04.419642 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:45:04.419735 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419711    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:45:04.419776 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:45:04.419814 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.419761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:45:04.420072 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.420058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-66ip9ogn7kmk9\""
Sep 16 18:45:04.432066 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.432042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54757d56d8-k6qw8"]
Sep 16 18:45:04.497213 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497213 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-grpc-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58nj\" (UniqueName: \"kubernetes.io/projected/c5976f70-bb7b-49e2-aca5-01ad137c495a-kube-api-access-j58nj\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497415 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5976f70-bb7b-49e2-aca5-01ad137c495a-metrics-client-ca\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.497568 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.497435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.598822 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.598780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.598822 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.598821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-grpc-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599003 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.598850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599003 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.598968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599118 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.599006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58nj\" (UniqueName: \"kubernetes.io/projected/c5976f70-bb7b-49e2-aca5-01ad137c495a-kube-api-access-j58nj\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599118 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.599037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599118 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.599075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5976f70-bb7b-49e2-aca5-01ad137c495a-metrics-client-ca\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.599118 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.599107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.600047 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.600015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5976f70-bb7b-49e2-aca5-01ad137c495a-metrics-client-ca\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.601575 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.601544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.601682 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.601669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.601956 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.601921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-grpc-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.602038 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.601984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.602114 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.602092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-tls\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.602253 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.602238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5976f70-bb7b-49e2-aca5-01ad137c495a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.608983 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.608960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58nj\" (UniqueName: \"kubernetes.io/projected/c5976f70-bb7b-49e2-aca5-01ad137c495a-kube-api-access-j58nj\") pod \"thanos-querier-54757d56d8-k6qw8\" (UID: \"c5976f70-bb7b-49e2-aca5-01ad137c495a\") " pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.726557 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.726473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:04.823370 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.823331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxcdf" event={"ID":"08ab8584-5ac2-4ab2-9bf9-007c943a0750","Type":"ContainerStarted","Data":"0a45e1d0df305ca41deff19cebbbaac61ca01903047c73ab064571a6f6902801"}
Sep 16 18:45:04.823718 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.823397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxcdf" event={"ID":"08ab8584-5ac2-4ab2-9bf9-007c943a0750","Type":"ContainerStarted","Data":"e13c57e5fcdc64dc16965661ec9dd7e428e441aa2c20911ae9e3cb8c757ef4e3"}
Sep 16 18:45:04.842129 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:04.842077    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fxcdf" podStartSLOduration=2.832441856 podStartE2EDuration="3.842061925s" totalImagesPullingTime="943.108832ms" totalInitContainerRuntime="66.511237ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:04.840766911 +0000 UTC m=+79.903106161" watchObservedRunningTime="2026-09-16 18:45:04.842061925 +0000 UTC m=+79.904401213"
Sep 16 18:45:05.000880 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:05.000802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54757d56d8-k6qw8"]
Sep 16 18:45:05.004078 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:45:05.004051    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5976f70_bb7b_49e2_aca5_01ad137c495a.slice/crio-a64dc6f40abeff068f4c9adbe43f6f4c1400756fe7a95dd17af1b15dd3a9a89a WatchSource:0}: Error finding container a64dc6f40abeff068f4c9adbe43f6f4c1400756fe7a95dd17af1b15dd3a9a89a: Status 404 returned error can't find the container with id a64dc6f40abeff068f4c9adbe43f6f4c1400756fe7a95dd17af1b15dd3a9a89a
Sep 16 18:45:05.829143 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:05.829060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"a64dc6f40abeff068f4c9adbe43f6f4c1400756fe7a95dd17af1b15dd3a9a89a"}
Sep 16 18:45:06.192678 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.192643    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"]
Sep 16 18:45:06.194863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.194832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.197991 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.197920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:45:06.198121 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.197936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4zgxt\""
Sep 16 18:45:06.208919 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.208896    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"]
Sep 16 18:45:06.315648 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.315609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.315863 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.315679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh789\" (UniqueName: \"kubernetes.io/projected/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-kube-api-access-qh789\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.416440 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.416393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh789\" (UniqueName: \"kubernetes.io/projected/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-kube-api-access-qh789\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.416629 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.416513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.419236 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.419210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.425050 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.424909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh789\" (UniqueName: \"kubernetes.io/projected/3db0cae2-5bdc-4727-bf51-09cc3d97bb47-kube-api-access-qh789\") pod \"monitoring-plugin-5d85778b9d-7wb68\" (UID: \"3db0cae2-5bdc-4727-bf51-09cc3d97bb47\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:06.506175 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:06.506080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:07.068305 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:07.068230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"]
Sep 16 18:45:07.071411 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:45:07.071383    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db0cae2_5bdc_4727_bf51_09cc3d97bb47.slice/crio-a5beed525ba66b51bc2cddb462cd87286db2bd6516259588486997ada2a0a109 WatchSource:0}: Error finding container a5beed525ba66b51bc2cddb462cd87286db2bd6516259588486997ada2a0a109: Status 404 returned error can't find the container with id a5beed525ba66b51bc2cddb462cd87286db2bd6516259588486997ada2a0a109
Sep 16 18:45:07.844347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:07.844184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"598b3dc1976042ecc9724995c029fef777a602a627ef79dd0dffe4cec664bf66"}
Sep 16 18:45:07.844347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:07.844280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"c36adf08c9ac0c5701de69db60aa1790134cec003632804ffb1e3f62dd39dea7"}
Sep 16 18:45:07.844347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:07.844325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"b50ddc3944549228d90218fc91506dc0e5086052373fc2d0d6bfdc139f75dd96"}
Sep 16 18:45:07.848614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:07.848577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68" event={"ID":"3db0cae2-5bdc-4727-bf51-09cc3d97bb47","Type":"ContainerStarted","Data":"a5beed525ba66b51bc2cddb462cd87286db2bd6516259588486997ada2a0a109"}
Sep 16 18:45:08.797181 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.797076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6pt88"
Sep 16 18:45:08.855654 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.855618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"76d04c979ec1b81bf0a88e41a4816ff2fba701626e6d97675b463fcff89aa1ad"}
Sep 16 18:45:08.855824 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.855697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"a374084f9c8b4ab02cc10339c0efafabec55dc20a0a7cda8af69e244abe981bb"}
Sep 16 18:45:08.855824 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.855737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" event={"ID":"c5976f70-bb7b-49e2-aca5-01ad137c495a","Type":"ContainerStarted","Data":"4f1d4c91873c553f715d069e2d0996b200f0e0f644e5b759ac0a3f80d442d83c"}
Sep 16 18:45:08.855824 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.855777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:08.857317 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.857284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68" event={"ID":"3db0cae2-5bdc-4727-bf51-09cc3d97bb47","Type":"ContainerStarted","Data":"348b05a400d4d06394b979cfcac2de9c22409935cec7ded6d86bf9aab1bba5a1"}
Sep 16 18:45:08.857453 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.857426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:08.862124 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.862099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68"
Sep 16 18:45:08.878800 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.878752    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8" podStartSLOduration=1.63468002 podStartE2EDuration="4.878736546s" totalImagesPullingTime="3.244056526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.876447858 +0000 UTC m=+83.938787108" watchObservedRunningTime="2026-09-16 18:45:08.878736546 +0000 UTC m=+83.941075796"
Sep 16 18:45:08.892778 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:08.892733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7wb68" podStartSLOduration=1.557197504 podStartE2EDuration="2.892717434s" totalImagesPullingTime="1.33551993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.891336068 +0000 UTC m=+83.953675329" watchObservedRunningTime="2026-09-16 18:45:08.892717434 +0000 UTC m=+83.955056729"
Sep 16 18:45:10.296696 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:10.296661    2581 patch_prober.go:28] interesting pod/image-registry-54b98bf9c8-9cdsp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:45:10.297055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:10.296713    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:45:10.692197 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:10.692145    2581 patch_prober.go:28] interesting pod/image-registry-f9bd6d57-bggql container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:45:10.692351 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:10.692220    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f9bd6d57-bggql" podUID="f62a0da4-c4e6-4c89-a9d7-6a3654803849" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:45:13.514508 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.514466    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-v4frq"]
Sep 16 18:45:13.517389 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.517366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:13.519762 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.519741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.519881 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.519845    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-d8pkn\""
Sep 16 18:45:13.519991 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.519978    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.526760 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.526738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-v4frq"]
Sep 16 18:45:13.674632 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.674594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24dw\" (UniqueName: \"kubernetes.io/projected/de8dde70-47db-41ee-9200-2352aedf2b65-kube-api-access-j24dw\") pod \"downloads-76f99584f4-v4frq\" (UID: \"de8dde70-47db-41ee-9200-2352aedf2b65\") " pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:13.756063 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.756035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f9bd6d57-bggql"
Sep 16 18:45:13.775418 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.775344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24dw\" (UniqueName: \"kubernetes.io/projected/de8dde70-47db-41ee-9200-2352aedf2b65-kube-api-access-j24dw\") pod \"downloads-76f99584f4-v4frq\" (UID: \"de8dde70-47db-41ee-9200-2352aedf2b65\") " pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:13.796456 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.796434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24dw\" (UniqueName: \"kubernetes.io/projected/de8dde70-47db-41ee-9200-2352aedf2b65-kube-api-access-j24dw\") pod \"downloads-76f99584f4-v4frq\" (UID: \"de8dde70-47db-41ee-9200-2352aedf2b65\") " pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:13.826182 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:13.826135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:14.100761 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:14.100729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-v4frq"]
Sep 16 18:45:14.104168 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:45:14.104126    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8dde70_47db_41ee_9200_2352aedf2b65.slice/crio-c4854591cde608fd96b6c9f5ded464c3ca7834d83607f47b06763d8dce3557fc WatchSource:0}: Error finding container c4854591cde608fd96b6c9f5ded464c3ca7834d83607f47b06763d8dce3557fc: Status 404 returned error can't find the container with id c4854591cde608fd96b6c9f5ded464c3ca7834d83607f47b06763d8dce3557fc
Sep 16 18:45:14.869795 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:14.869760    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54757d56d8-k6qw8"
Sep 16 18:45:14.885061 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:14.885027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-v4frq" event={"ID":"de8dde70-47db-41ee-9200-2352aedf2b65","Type":"ContainerStarted","Data":"c4854591cde608fd96b6c9f5ded464c3ca7834d83607f47b06763d8dce3557fc"}
Sep 16 18:45:15.310219 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.310119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry" containerID="cri-o://f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d" gracePeriod=30
Sep 16 18:45:15.588561 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.588532    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:45:15.690774 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.690939 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.690939 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vzhc\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.690939 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.691055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.691055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.690991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.691055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.691016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.691055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.691051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token\") pod \"ff50c590-3f4e-464d-8401-4e513656b4aa\" (UID: \"ff50c590-3f4e-464d-8401-4e513656b4aa\") "
Sep 16 18:45:15.691776 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.691704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:15.692070 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.692021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:15.694952 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.694924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:15.695780 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.695539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:15.695780 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.695672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:15.695780 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.695724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:15.696001 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.695834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc" (OuterVolumeSpecName: "kube-api-access-2vzhc") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "kube-api-access-2vzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:15.702030 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.701995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ff50c590-3f4e-464d-8401-4e513656b4aa" (UID: "ff50c590-3f4e-464d-8401-4e513656b4aa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:15.792280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792245    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff50c590-3f4e-464d-8401-4e513656b4aa-ca-trust-extracted\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792279    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-image-registry-private-configuration\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792280 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792290    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzhc\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-kube-api-access-2vzhc\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792304    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff50c590-3f4e-464d-8401-4e513656b4aa-installation-pull-secrets\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792314    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792324    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-trusted-ca\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792337    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff50c590-3f4e-464d-8401-4e513656b4aa-registry-certificates\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.792539 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.792350    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff50c590-3f4e-464d-8401-4e513656b4aa-bound-sa-token\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:45:15.893371 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.893336    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerID="f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d" exitCode=0
Sep 16 18:45:15.893768 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.893398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp"
Sep 16 18:45:15.893768 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.893420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" event={"ID":"ff50c590-3f4e-464d-8401-4e513656b4aa","Type":"ContainerDied","Data":"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"}
Sep 16 18:45:15.893768 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.893492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b98bf9c8-9cdsp" event={"ID":"ff50c590-3f4e-464d-8401-4e513656b4aa","Type":"ContainerDied","Data":"d645f49bb8e8419e30f21c496e78aeb8231964df3a943ec0ecc83b353d3c475d"}
Sep 16 18:45:15.893768 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.893520    2581 scope.go:117] "RemoveContainer" containerID="f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"
Sep 16 18:45:15.915649 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.915620    2581 scope.go:117] "RemoveContainer" containerID="f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"
Sep 16 18:45:15.916195 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:45:15.916143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d\": container with ID starting with f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d not found: ID does not exist" containerID="f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"
Sep 16 18:45:15.916317 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.916207    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d"} err="failed to get container status \"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d\": rpc error: code = NotFound desc = could not find container \"f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d\": container with ID starting with f66fdeb99be0ae04ee177b47dab9e0bc5afca1084f9670058554f5a92e84890d not found: ID does not exist"
Sep 16 18:45:15.930060 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.929405    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:45:15.933875 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:15.933847    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54b98bf9c8-9cdsp"]
Sep 16 18:45:17.415286 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:17.415251    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" path="/var/lib/kubelet/pods/ff50c590-3f4e-464d-8401-4e513656b4aa/volumes"
Sep 16 18:45:18.974274 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.974241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:45:18.974669 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.974650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry"
Sep 16 18:45:18.974729 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.974670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry"
Sep 16 18:45:18.974772 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.974759    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff50c590-3f4e-464d-8401-4e513656b4aa" containerName="registry"
Sep 16 18:45:18.977605 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.977577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:18.980890 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.980820    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jm68g\""
Sep 16 18:45:18.981030 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.980893    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:45:18.981030 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.980921    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:45:18.981030 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.980945    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:45:18.981030 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.980893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:45:18.981387 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.981359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:45:18.991258 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:18.991235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:45:19.119947 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.119907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.120122 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.119999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.120122 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.120083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.120255 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.120131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.120255 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.120179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.120255 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.120247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckmw\" (UniqueName: \"kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221478 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221478 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckmw\" (UniqueName: \"kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.221728 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.221657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.222484 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.222410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.222484 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.222411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.223036 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.223011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.224803 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.224727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.225208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.225189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.230594 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.230568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckmw\" (UniqueName: \"kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw\") pod \"console-66767b4dcd-6p265\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") " pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.290015 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.289977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:19.571347 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.571261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:45:19.573919 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:45:19.573887    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd815dc_a4e7_4392_8705_4545b036a103.slice/crio-27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c WatchSource:0}: Error finding container 27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c: Status 404 returned error can't find the container with id 27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c
Sep 16 18:45:19.914294 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:19.914256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66767b4dcd-6p265" event={"ID":"7bd815dc-a4e7-4392-8705-4545b036a103","Type":"ContainerStarted","Data":"27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c"}
Sep 16 18:45:22.748170 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:22.748072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h7gn6"
Sep 16 18:45:23.933865 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:23.933823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66767b4dcd-6p265" event={"ID":"7bd815dc-a4e7-4392-8705-4545b036a103","Type":"ContainerStarted","Data":"c440d7efad25c4c16ecc3635b93ae9bd1823fa56843c3ccb5c2d589b980b9686"}
Sep 16 18:45:23.951657 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:23.951593    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66767b4dcd-6p265" podStartSLOduration=2.598312726 podStartE2EDuration="5.951567894s" totalImagesPullingTime="3.353255168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:23.950838889 +0000 UTC m=+99.013178141" watchObservedRunningTime="2026-09-16 18:45:23.951567894 +0000 UTC m=+99.013907144"
Sep 16 18:45:29.290696 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:29.290647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:29.291264 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:29.290746    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:29.296288 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:29.296266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:29.965447 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:29.965416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:45:31.971606 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:31.971554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-v4frq" event={"ID":"de8dde70-47db-41ee-9200-2352aedf2b65","Type":"ContainerStarted","Data":"a08c2d2eb1161cc202ec6409b16f83d3e5d049a620fd11d0b72297d69e71e4bd"}
Sep 16 18:45:31.972055 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:31.971714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:31.973383 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:31.973355    2581 patch_prober.go:28] interesting pod/downloads-76f99584f4-v4frq container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused" start-of-body=
Sep 16 18:45:31.973444 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:31.973410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-v4frq" podUID="de8dde70-47db-41ee-9200-2352aedf2b65" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 16 18:45:31.988935 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:31.988836    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-v4frq" podStartSLOduration=1.358529739 podStartE2EDuration="18.988814656s" totalImagesPullingTime="17.630284917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:31.987461709 +0000 UTC m=+107.049801168" watchObservedRunningTime="2026-09-16 18:45:31.988814656 +0000 UTC m=+107.051153908"
Sep 16 18:45:32.994260 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:32.994222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-v4frq"
Sep 16 18:45:39.819768 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:39.819729    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:45:47.036651 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:47.036562    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc51e700-c58d-48ce-b099-e448440ebf32" containerID="ffa220c3b7b6c1040c860afff0afdc45c8d5bea778883e0c5ef800a5c6407c9c" exitCode=0
Sep 16 18:45:47.036651 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:47.036635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j" event={"ID":"cc51e700-c58d-48ce-b099-e448440ebf32","Type":"ContainerDied","Data":"ffa220c3b7b6c1040c860afff0afdc45c8d5bea778883e0c5ef800a5c6407c9c"}
Sep 16 18:45:47.037234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:47.037093    2581 scope.go:117] "RemoveContainer" containerID="ffa220c3b7b6c1040c860afff0afdc45c8d5bea778883e0c5ef800a5c6407c9c"
Sep 16 18:45:48.044914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:45:48.044876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lhw7j" event={"ID":"cc51e700-c58d-48ce-b099-e448440ebf32","Type":"ContainerStarted","Data":"aed0eaea9f8702efc35d87169aadd3bf91eb390af2e1c437455828bb481f0285"}
Sep 16 18:46:04.842672 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:04.842626    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66767b4dcd-6p265" podUID="7bd815dc-a4e7-4392-8705-4545b036a103" containerName="console" containerID="cri-o://c440d7efad25c4c16ecc3635b93ae9bd1823fa56843c3ccb5c2d589b980b9686" gracePeriod=15
Sep 16 18:46:05.111124 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.111095    2581 generic.go:356] "Generic (PLEG): container finished" podID="7bd815dc-a4e7-4392-8705-4545b036a103" containerID="c440d7efad25c4c16ecc3635b93ae9bd1823fa56843c3ccb5c2d589b980b9686" exitCode=0
Sep 16 18:46:05.111259 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.111180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66767b4dcd-6p265" event={"ID":"7bd815dc-a4e7-4392-8705-4545b036a103","Type":"ContainerDied","Data":"c440d7efad25c4c16ecc3635b93ae9bd1823fa56843c3ccb5c2d589b980b9686"}
Sep 16 18:46:05.111259 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.111223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66767b4dcd-6p265" event={"ID":"7bd815dc-a4e7-4392-8705-4545b036a103","Type":"ContainerDied","Data":"27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c"}
Sep 16 18:46:05.111259 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.111237    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27a6d6f051a4b2e47b85b99b46eacd58d17eb1fa5a578b906cb8decb4fdd462c"
Sep 16 18:46:05.115348 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.115327    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:46:05.205140 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205322 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205322 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205322 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205484 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205484 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckmw\" (UniqueName: \"kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw\") pod \"7bd815dc-a4e7-4392-8705-4545b036a103\" (UID: \"7bd815dc-a4e7-4392-8705-4545b036a103\") "
Sep 16 18:46:05.205620 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:05.205756 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca" (OuterVolumeSpecName: "service-ca") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:05.205825 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.205782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config" (OuterVolumeSpecName: "console-config") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:05.207597 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.207564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw" (OuterVolumeSpecName: "kube-api-access-9ckmw") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "kube-api-access-9ckmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:05.207689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.207607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:05.207689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.207623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7bd815dc-a4e7-4392-8705-4545b036a103" (UID: "7bd815dc-a4e7-4392-8705-4545b036a103"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:05.306584 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306541    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-service-ca\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:05.306584 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306576    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-console-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:05.306584 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306590    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:05.306816 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306602    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckmw\" (UniqueName: \"kubernetes.io/projected/7bd815dc-a4e7-4392-8705-4545b036a103-kube-api-access-9ckmw\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:05.306816 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306614    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd815dc-a4e7-4392-8705-4545b036a103-console-oauth-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:05.306816 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:05.306626    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd815dc-a4e7-4392-8705-4545b036a103-oauth-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.115208 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:06.115145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66767b4dcd-6p265"
Sep 16 18:46:06.131004 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:06.130972    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:46:06.134834 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:06.134807    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66767b4dcd-6p265"]
Sep 16 18:46:07.413875 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:07.413831    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd815dc-a4e7-4392-8705-4545b036a103" path="/var/lib/kubelet/pods/7bd815dc-a4e7-4392-8705-4545b036a103/volumes"
Sep 16 18:46:29.163395 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.163346    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:46:29.163831 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.163644    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd815dc-a4e7-4392-8705-4545b036a103" containerName="console"
Sep 16 18:46:29.163831 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.163657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd815dc-a4e7-4392-8705-4545b036a103" containerName="console"
Sep 16 18:46:29.163831 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.163712    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd815dc-a4e7-4392-8705-4545b036a103" containerName="console"
Sep 16 18:46:29.165640 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.165624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.168007 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.167982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jm68g\""
Sep 16 18:46:29.168146 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.167982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:46:29.168679 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.168661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:46:29.168733 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.168688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:46:29.168733 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.168716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:46:29.169537 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.169515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:46:29.172613 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.172596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:46:29.175840 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.175820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:46:29.287882 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.287843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp77x\" (UniqueName: \"kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.287889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.287974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.288012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288074 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.288038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288271 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.288077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.288271 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.288176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.388820 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.388786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.388973 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.388834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.388973 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.388860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp77x\" (UniqueName: \"kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389085 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.388984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389085 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389085 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389261 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389701 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389831 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389831 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.389966 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.389822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.391451 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.391432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.391534 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.391456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.398311 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.398287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp77x\" (UniqueName: \"kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x\") pod \"console-5cd6d99578-8zgxn\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") " pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.474810 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.474730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:29.747056 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:29.746961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:46:29.751181 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:46:29.751130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afadeb8_3479_432c_9be5_9ca899c90cbe.slice/crio-3ce0c2e2fc7bafbb864753517339fb98225bc7c4ec706c59338a51ce96b493ae WatchSource:0}: Error finding container 3ce0c2e2fc7bafbb864753517339fb98225bc7c4ec706c59338a51ce96b493ae: Status 404 returned error can't find the container with id 3ce0c2e2fc7bafbb864753517339fb98225bc7c4ec706c59338a51ce96b493ae
Sep 16 18:46:30.209128 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:30.208904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6d99578-8zgxn" event={"ID":"1afadeb8-3479-432c-9be5-9ca899c90cbe","Type":"ContainerStarted","Data":"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"}
Sep 16 18:46:30.209128 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:30.208969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6d99578-8zgxn" event={"ID":"1afadeb8-3479-432c-9be5-9ca899c90cbe","Type":"ContainerStarted","Data":"3ce0c2e2fc7bafbb864753517339fb98225bc7c4ec706c59338a51ce96b493ae"}
Sep 16 18:46:30.226449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:30.226393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd6d99578-8zgxn" podStartSLOduration=1.22637671 podStartE2EDuration="1.22637671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:30.22538506 +0000 UTC m=+165.287724311" watchObservedRunningTime="2026-09-16 18:46:30.22637671 +0000 UTC m=+165.288715958"
Sep 16 18:46:39.474971 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:39.474875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:39.474971 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:39.474949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:39.479534 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:39.479511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:46:40.248816 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:46:40.248787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:47:40.048296 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.048251    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:47:40.053953 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.053901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.062861 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.062836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:47:40.106758 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106758 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106977 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106977 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106977 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106977 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2q9\" (UniqueName: \"kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.106977 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.106912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208177 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208177 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208663 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2q9\" (UniqueName: \"kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.208663 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.208467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.209061 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.209030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.209210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.209030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.209210 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.209196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.209364 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.209338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.210680 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.210662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.210793 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.210776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.216860 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.216842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2q9\" (UniqueName: \"kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9\") pod \"console-67cfbdf9c6-hs8j6\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") " pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.364758 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.364720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:40.637256 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:40.637219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:47:40.639957 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:47:40.639926    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbca215f_0c1d_47eb_a5b1_514ed6332a1b.slice/crio-ef06aa5b3266efc250e0bfa67364053a3bf8546233169f074581e5aa53af91f6 WatchSource:0}: Error finding container ef06aa5b3266efc250e0bfa67364053a3bf8546233169f074581e5aa53af91f6: Status 404 returned error can't find the container with id ef06aa5b3266efc250e0bfa67364053a3bf8546233169f074581e5aa53af91f6
Sep 16 18:47:41.469655 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:41.469616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cfbdf9c6-hs8j6" event={"ID":"bbca215f-0c1d-47eb-a5b1-514ed6332a1b","Type":"ContainerStarted","Data":"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"}
Sep 16 18:47:41.470042 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:41.469670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cfbdf9c6-hs8j6" event={"ID":"bbca215f-0c1d-47eb-a5b1-514ed6332a1b","Type":"ContainerStarted","Data":"ef06aa5b3266efc250e0bfa67364053a3bf8546233169f074581e5aa53af91f6"}
Sep 16 18:47:41.500599 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:41.500537    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67cfbdf9c6-hs8j6" podStartSLOduration=1.500516832 podStartE2EDuration="1.500516832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.500517721 +0000 UTC m=+236.562856971" watchObservedRunningTime="2026-09-16 18:47:41.500516832 +0000 UTC m=+236.562856095"
Sep 16 18:47:50.365637 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:50.365550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:50.366104 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:50.365667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:50.370318 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:50.370291    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:50.507234 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:50.507209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:47:50.556197 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:47:50.556166    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:48:15.580908 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.580784    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd6d99578-8zgxn" podUID="1afadeb8-3479-432c-9be5-9ca899c90cbe" containerName="console" containerID="cri-o://2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856" gracePeriod=15
Sep 16 18:48:15.817473 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.817448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:48:15.892835 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.892802    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.892835 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.892837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.892862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.892881    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.892896    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp77x\" (UniqueName: \"kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893073 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle\") pod \"1afadeb8-3479-432c-9be5-9ca899c90cbe\" (UID: \"1afadeb8-3479-432c-9be5-9ca899c90cbe\") "
Sep 16 18:48:15.893365 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893275    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca" (OuterVolumeSpecName: "service-ca") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:15.893417 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893363    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config" (OuterVolumeSpecName: "console-config") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:15.893668 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:15.893668 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.893647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:15.895144 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.895114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x" (OuterVolumeSpecName: "kube-api-access-fp77x") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "kube-api-access-fp77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:15.895250 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.895137    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:15.895250 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.895177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1afadeb8-3479-432c-9be5-9ca899c90cbe" (UID: "1afadeb8-3479-432c-9be5-9ca899c90cbe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:15.993594 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993566    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-service-ca\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993594 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993590    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-oauth-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993594 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993600    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-oauth-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993815 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993611    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993815 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993621    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-console-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993815 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993632    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp77x\" (UniqueName: \"kubernetes.io/projected/1afadeb8-3479-432c-9be5-9ca899c90cbe-kube-api-access-fp77x\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:15.993815 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:15.993643    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1afadeb8-3479-432c-9be5-9ca899c90cbe-trusted-ca-bundle\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.600760 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.600725    2581 generic.go:356] "Generic (PLEG): container finished" podID="1afadeb8-3479-432c-9be5-9ca899c90cbe" containerID="2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856" exitCode=0
Sep 16 18:48:16.601198 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.600768    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6d99578-8zgxn"
Sep 16 18:48:16.601198 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.600802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6d99578-8zgxn" event={"ID":"1afadeb8-3479-432c-9be5-9ca899c90cbe","Type":"ContainerDied","Data":"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"}
Sep 16 18:48:16.601198 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.600847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6d99578-8zgxn" event={"ID":"1afadeb8-3479-432c-9be5-9ca899c90cbe","Type":"ContainerDied","Data":"3ce0c2e2fc7bafbb864753517339fb98225bc7c4ec706c59338a51ce96b493ae"}
Sep 16 18:48:16.601198 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.600867    2581 scope.go:117] "RemoveContainer" containerID="2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"
Sep 16 18:48:16.609388 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.609373    2581 scope.go:117] "RemoveContainer" containerID="2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"
Sep 16 18:48:16.609644 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:48:16.609628    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856\": container with ID starting with 2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856 not found: ID does not exist" containerID="2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"
Sep 16 18:48:16.609697 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.609650    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856"} err="failed to get container status \"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856\": rpc error: code = NotFound desc = could not find container \"2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856\": container with ID starting with 2fc4aa766fef6eff4fdd43e386ce1bae598144af656af2d8bfd4ec38d61c1856 not found: ID does not exist"
Sep 16 18:48:16.620141 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.620117    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:48:16.622014 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:16.621994    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd6d99578-8zgxn"]
Sep 16 18:48:17.413482 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:17.413449    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1afadeb8-3479-432c-9be5-9ca899c90cbe" path="/var/lib/kubelet/pods/1afadeb8-3479-432c-9be5-9ca899c90cbe/volumes"
Sep 16 18:48:45.296753 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:48:45.296723    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:16.643212 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.643177    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"]
Sep 16 18:49:16.644500 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.643460    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afadeb8-3479-432c-9be5-9ca899c90cbe" containerName="console"
Sep 16 18:49:16.644500 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.643470    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afadeb8-3479-432c-9be5-9ca899c90cbe" containerName="console"
Sep 16 18:49:16.644500 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.643525    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1afadeb8-3479-432c-9be5-9ca899c90cbe" containerName="console"
Sep 16 18:49:16.645042 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.645018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.648046 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.648007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:49:16.648046 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.648039    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:49:16.648229 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.648067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-98562\""
Sep 16 18:49:16.648229 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.648172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:49:16.663323 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.663297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"]
Sep 16 18:49:16.753159 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.753121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7qd\" (UniqueName: \"kubernetes.io/projected/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-kube-api-access-4k7qd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.753356 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.753276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.853915 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.853882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.854092 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.853933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7qd\" (UniqueName: \"kubernetes.io/projected/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-kube-api-access-4k7qd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.856382 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.856358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.863325 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.863301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7qd\" (UniqueName: \"kubernetes.io/projected/2701233a-3a3f-4e84-bbdf-1e970cf7cac9-kube-api-access-4k7qd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpmfz\" (UID: \"2701233a-3a3f-4e84-bbdf-1e970cf7cac9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:16.955058 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:16.954964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:17.229822 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:17.229722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"]
Sep 16 18:49:17.233211 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:49:17.233179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2701233a_3a3f_4e84_bbdf_1e970cf7cac9.slice/crio-e9f97032b2d5c4431c9f0db86554636ff83e363c3524c8aabc12e15242053e0e WatchSource:0}: Error finding container e9f97032b2d5c4431c9f0db86554636ff83e363c3524c8aabc12e15242053e0e: Status 404 returned error can't find the container with id e9f97032b2d5c4431c9f0db86554636ff83e363c3524c8aabc12e15242053e0e
Sep 16 18:49:17.235034 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:17.235014    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:17.835395 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:17.835361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz" event={"ID":"2701233a-3a3f-4e84-bbdf-1e970cf7cac9","Type":"ContainerStarted","Data":"e9f97032b2d5c4431c9f0db86554636ff83e363c3524c8aabc12e15242053e0e"}
Sep 16 18:49:20.850994 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:20.850952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz" event={"ID":"2701233a-3a3f-4e84-bbdf-1e970cf7cac9","Type":"ContainerStarted","Data":"2ce10ef603fb6a0ebee0b0750dc9bae8a1a827812de653d9ccb3e1af69b5aa28"}
Sep 16 18:49:20.851471 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:20.851031    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:20.869369 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:20.869310    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz" podStartSLOduration=1.662543141 podStartE2EDuration="4.869292423s" totalImagesPullingTime="3.206749282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:20.867983285 +0000 UTC m=+335.930322527" watchObservedRunningTime="2026-09-16 18:49:20.869292423 +0000 UTC m=+335.931631674"
Sep 16 18:49:21.631868 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.631833    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"]
Sep 16 18:49:21.633916 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.633898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.636130 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.636107    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:49:21.636275 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.636200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-qvp47\""
Sep 16 18:49:21.636275 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.636201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:49:21.643932 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.643906    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"]
Sep 16 18:49:21.796435 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.796394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ld2\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-kube-api-access-n4ld2\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.796609 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.796441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.796660 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.796605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/514aaaa7-ef3a-4850-b368-e15912c0c7ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.897323 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.897238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/514aaaa7-ef3a-4850-b368-e15912c0c7ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.897323 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.897283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ld2\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-kube-api-access-n4ld2\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.897323 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.897310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.897820 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:49:21.897416    2581 secret.go:281] references non-existent secret key: tls.crt
Sep 16 18:49:21.897820 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:49:21.897429    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 18:49:21.897820 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:49:21.897445    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf: references non-existent secret key: tls.crt
Sep 16 18:49:21.897820 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:49:21.897496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates podName:514aaaa7-ef3a-4850-b368-e15912c0c7ea nodeName:}" failed. No retries permitted until 2026-09-16 18:49:22.39747966 +0000 UTC m=+337.459818908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates") pod "keda-metrics-apiserver-77857b7b98-sn6wf" (UID: "514aaaa7-ef3a-4850-b368-e15912c0c7ea") : references non-existent secret key: tls.crt
Sep 16 18:49:21.897820 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.897695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/514aaaa7-ef3a-4850-b368-e15912c0c7ea-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:21.905614 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:21.905589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ld2\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-kube-api-access-n4ld2\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:22.254938 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.254851    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cg247"]
Sep 16 18:49:22.257065 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.257050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.259029 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.259009    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:49:22.266716 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.266694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cg247"]
Sep 16 18:49:22.402185 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.402132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-certificates\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.402348 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.402250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:22.402348 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.402312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5nx\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-kube-api-access-kh5nx\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.406742 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.406713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/514aaaa7-ef3a-4850-b368-e15912c0c7ea-certificates\") pod \"keda-metrics-apiserver-77857b7b98-sn6wf\" (UID: \"514aaaa7-ef3a-4850-b368-e15912c0c7ea\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:22.503588 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.503544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-certificates\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.503733 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.503657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5nx\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-kube-api-access-kh5nx\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.506670 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.506588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-certificates\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.514492 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.514456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5nx\" (UniqueName: \"kubernetes.io/projected/0f4ce8a8-70f1-414b-bd86-82facdfa40f6-kube-api-access-kh5nx\") pod \"keda-admission-6c4d864b86-cg247\" (UID: \"0f4ce8a8-70f1-414b-bd86-82facdfa40f6\") " pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.544263 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.544231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:22.567708 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.567671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:22.825725 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.825593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"]
Sep 16 18:49:22.829210 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:49:22.829183    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514aaaa7_ef3a_4850_b368_e15912c0c7ea.slice/crio-13ee42b0b69a550d6c048760db9e3b3a3ee6b88bbcaa70c71ccb0f81dc6a06bc WatchSource:0}: Error finding container 13ee42b0b69a550d6c048760db9e3b3a3ee6b88bbcaa70c71ccb0f81dc6a06bc: Status 404 returned error can't find the container with id 13ee42b0b69a550d6c048760db9e3b3a3ee6b88bbcaa70c71ccb0f81dc6a06bc
Sep 16 18:49:22.861529 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.861495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf" event={"ID":"514aaaa7-ef3a-4850-b368-e15912c0c7ea","Type":"ContainerStarted","Data":"13ee42b0b69a550d6c048760db9e3b3a3ee6b88bbcaa70c71ccb0f81dc6a06bc"}
Sep 16 18:49:22.861914 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:22.861893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cg247"]
Sep 16 18:49:22.865062 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:49:22.865035    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4ce8a8_70f1_414b_bd86_82facdfa40f6.slice/crio-87117e7eee66852994cc4b5adf796084f09c7194a3fcb32a16e40e384cc11fab WatchSource:0}: Error finding container 87117e7eee66852994cc4b5adf796084f09c7194a3fcb32a16e40e384cc11fab: Status 404 returned error can't find the container with id 87117e7eee66852994cc4b5adf796084f09c7194a3fcb32a16e40e384cc11fab
Sep 16 18:49:23.869352 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:23.869250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-cg247" event={"ID":"0f4ce8a8-70f1-414b-bd86-82facdfa40f6","Type":"ContainerStarted","Data":"87117e7eee66852994cc4b5adf796084f09c7194a3fcb32a16e40e384cc11fab"}
Sep 16 18:49:24.875260 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:24.875220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-cg247" event={"ID":"0f4ce8a8-70f1-414b-bd86-82facdfa40f6","Type":"ContainerStarted","Data":"756199cad50b39659547e0dfeb2a3d15751f85e0d6f127336261ec30caa43ea5"}
Sep 16 18:49:24.875683 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:24.875334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:49:24.895568 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:24.895513    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-cg247" podStartSLOduration=1.35991906 podStartE2EDuration="2.895494917s" totalImagesPullingTime="1.535575857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:24.894036886 +0000 UTC m=+339.956376138" watchObservedRunningTime="2026-09-16 18:49:24.895494917 +0000 UTC m=+339.957834161"
Sep 16 18:49:25.882526 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:25.882489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf" event={"ID":"514aaaa7-ef3a-4850-b368-e15912c0c7ea","Type":"ContainerStarted","Data":"a949e37bf62e224c14930708e08cbe293078386034453f26dc2f542d0bd4c7df"}
Sep 16 18:49:25.882981 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:25.882648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:25.898903 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:25.898840    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf" podStartSLOduration=2.313047731 podStartE2EDuration="4.898817877s" totalImagesPullingTime="2.585770146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:25.897365491 +0000 UTC m=+340.959704741" watchObservedRunningTime="2026-09-16 18:49:25.898817877 +0000 UTC m=+340.961157131"
Sep 16 18:49:36.891875 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:36.891846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-sn6wf"
Sep 16 18:49:41.857575 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:41.857494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpmfz"
Sep 16 18:49:45.885862 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:49:45.885823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-cg247"
Sep 16 18:50:24.188071 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.188033    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ffddb9bf5-j5nqw"]
Sep 16 18:50:24.191449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.191428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.211438 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.211410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffddb9bf5-j5nqw"]
Sep 16 18:50:24.310164 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-service-ca\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310336 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310336 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-console-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310336 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-oauth-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-oauth-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-trusted-ca-bundle\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.310441 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.310374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9hq\" (UniqueName: \"kubernetes.io/projected/03e36c06-6084-484d-839b-7f3f276fe9d9-kube-api-access-qw9hq\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411203 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-console-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411203 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-oauth-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-oauth-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-trusted-ca-bundle\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9hq\" (UniqueName: \"kubernetes.io/projected/03e36c06-6084-484d-839b-7f3f276fe9d9-kube-api-access-qw9hq\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-service-ca\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411462 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.411984 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.411956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-console-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.412109 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.412092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-oauth-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.412251 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.412229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-service-ca\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.412299 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.412286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e36c06-6084-484d-839b-7f3f276fe9d9-trusted-ca-bundle\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.414282 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.414263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-serving-cert\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.414384 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.414315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e36c06-6084-484d-839b-7f3f276fe9d9-console-oauth-config\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.420004 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.419982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9hq\" (UniqueName: \"kubernetes.io/projected/03e36c06-6084-484d-839b-7f3f276fe9d9-kube-api-access-qw9hq\") pod \"console-7ffddb9bf5-j5nqw\" (UID: \"03e36c06-6084-484d-839b-7f3f276fe9d9\") " pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.500655 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.500575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:24.772178 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:24.772084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffddb9bf5-j5nqw"]
Sep 16 18:50:24.775501 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:50:24.775470    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e36c06_6084_484d_839b_7f3f276fe9d9.slice/crio-810fc6e30e62dec1d8cf21646e4914de9dff81bb2c0d6d22e9cc630a88057436 WatchSource:0}: Error finding container 810fc6e30e62dec1d8cf21646e4914de9dff81bb2c0d6d22e9cc630a88057436: Status 404 returned error can't find the container with id 810fc6e30e62dec1d8cf21646e4914de9dff81bb2c0d6d22e9cc630a88057436
Sep 16 18:50:25.134138 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:25.134096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffddb9bf5-j5nqw" event={"ID":"03e36c06-6084-484d-839b-7f3f276fe9d9","Type":"ContainerStarted","Data":"35324a6bca66ddf10b7b08e18c0cfa870537d3f757b28ca3b1796ed815d5234e"}
Sep 16 18:50:25.134359 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:25.134167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffddb9bf5-j5nqw" event={"ID":"03e36c06-6084-484d-839b-7f3f276fe9d9","Type":"ContainerStarted","Data":"810fc6e30e62dec1d8cf21646e4914de9dff81bb2c0d6d22e9cc630a88057436"}
Sep 16 18:50:25.152933 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:25.152878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ffddb9bf5-j5nqw" podStartSLOduration=1.152862724 podStartE2EDuration="1.152862724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:25.152224424 +0000 UTC m=+400.214563687" watchObservedRunningTime="2026-09-16 18:50:25.152862724 +0000 UTC m=+400.215202029"
Sep 16 18:50:34.501285 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:34.501224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:34.501689 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:34.501305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:34.506170 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:34.506132    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:35.180169 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:35.180126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ffddb9bf5-j5nqw"
Sep 16 18:50:35.226792 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:35.226748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:50:38.610556 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.610521    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-l9fj9"]
Sep 16 18:50:38.614005 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.613980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.616456 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.616430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:50:38.616557 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.616430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:50:38.617098 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.617075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:50:38.617226 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.617130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 18:50:38.622847 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.622823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-l9fj9"]
Sep 16 18:50:38.623144 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.623123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lcj\" (UniqueName: \"kubernetes.io/projected/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-kube-api-access-z8lcj\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.623254 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.623231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-data\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.724648 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.724611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-data\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.724821 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.724680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lcj\" (UniqueName: \"kubernetes.io/projected/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-kube-api-access-z8lcj\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.724999 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.724978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-data\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.732069 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.732042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lcj\" (UniqueName: \"kubernetes.io/projected/859c5c3f-a95d-4dcf-91ad-ab500a24f07f-kube-api-access-z8lcj\") pod \"seaweedfs-64568bcd49-l9fj9\" (UID: \"859c5c3f-a95d-4dcf-91ad-ab500a24f07f\") " pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:38.923743 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:38.923651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:39.191169 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:39.191072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-l9fj9"]
Sep 16 18:50:39.194218 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:50:39.194185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859c5c3f_a95d_4dcf_91ad_ab500a24f07f.slice/crio-14623cfc5da34c106827dc4f0b1c3fcd54c7b94f0028b401ffed5c4031b71414 WatchSource:0}: Error finding container 14623cfc5da34c106827dc4f0b1c3fcd54c7b94f0028b401ffed5c4031b71414: Status 404 returned error can't find the container with id 14623cfc5da34c106827dc4f0b1c3fcd54c7b94f0028b401ffed5c4031b71414
Sep 16 18:50:40.200786 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:40.200747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-l9fj9" event={"ID":"859c5c3f-a95d-4dcf-91ad-ab500a24f07f","Type":"ContainerStarted","Data":"14623cfc5da34c106827dc4f0b1c3fcd54c7b94f0028b401ffed5c4031b71414"}
Sep 16 18:50:42.210883 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:42.210842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-l9fj9" event={"ID":"859c5c3f-a95d-4dcf-91ad-ab500a24f07f","Type":"ContainerStarted","Data":"f9cded1995a286a0e4620dcd422099d64bb6a7fcd5b5e0b718625bd05e74bbc9"}
Sep 16 18:50:42.211293 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:42.210986    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:50:42.226652 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:42.226599    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-l9fj9" podStartSLOduration=1.329192995 podStartE2EDuration="4.226583426s" totalImagesPullingTime="2.897390431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:42.224458729 +0000 UTC m=+417.286797979" watchObservedRunningTime="2026-09-16 18:50:42.226583426 +0000 UTC m=+417.288922677"
Sep 16 18:50:48.217360 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:50:48.217322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-l9fj9"
Sep 16 18:51:00.252420 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.252329    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67cfbdf9c6-hs8j6" podUID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" containerName="console" containerID="cri-o://abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3" gracePeriod=15
Sep 16 18:51:00.493821 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.493793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:51:00.607285 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v2q9\" (UniqueName: \"kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607468 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607427    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle\") pod \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\" (UID: \"bbca215f-0c1d-47eb-a5b1-514ed6332a1b\") "
Sep 16 18:51:00.607779 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:00.607927 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca" (OuterVolumeSpecName: "service-ca") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:00.607993 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607942    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config" (OuterVolumeSpecName: "console-config") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:00.608050 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.607990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:00.609529 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.609503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9" (OuterVolumeSpecName: "kube-api-access-7v2q9") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "kube-api-access-7v2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:00.609874 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.609852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:00.609874 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.609865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bbca215f-0c1d-47eb-a5b1-514ed6332a1b" (UID: "bbca215f-0c1d-47eb-a5b1-514ed6332a1b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:00.708449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708402    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2q9\" (UniqueName: \"kubernetes.io/projected/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-kube-api-access-7v2q9\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708444    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708454    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-trusted-ca-bundle\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708449 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708463    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-oauth-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708713 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708472    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-oauth-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708713 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708480    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-console-serving-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:00.708713 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:00.708489    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bbca215f-0c1d-47eb-a5b1-514ed6332a1b-service-ca\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 18:51:01.295058 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.295025    2581 generic.go:356] "Generic (PLEG): container finished" podID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" containerID="abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3" exitCode=0
Sep 16 18:51:01.295482 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.295070    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67cfbdf9c6-hs8j6"
Sep 16 18:51:01.295482 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.295095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cfbdf9c6-hs8j6" event={"ID":"bbca215f-0c1d-47eb-a5b1-514ed6332a1b","Type":"ContainerDied","Data":"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"}
Sep 16 18:51:01.295482 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.295184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67cfbdf9c6-hs8j6" event={"ID":"bbca215f-0c1d-47eb-a5b1-514ed6332a1b","Type":"ContainerDied","Data":"ef06aa5b3266efc250e0bfa67364053a3bf8546233169f074581e5aa53af91f6"}
Sep 16 18:51:01.295482 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.295205    2581 scope.go:117] "RemoveContainer" containerID="abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"
Sep 16 18:51:01.303811 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.303788    2581 scope.go:117] "RemoveContainer" containerID="abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"
Sep 16 18:51:01.304133 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:51:01.304114    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3\": container with ID starting with abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3 not found: ID does not exist" containerID="abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"
Sep 16 18:51:01.304287 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.304142    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3"} err="failed to get container status \"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3\": rpc error: code = NotFound desc = could not find container \"abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3\": container with ID starting with abe73bfa90adbe6c689743cbd9714aa1bd9b4fe45e6ca8ebedfefbf2bb3875d3 not found: ID does not exist"
Sep 16 18:51:01.317354 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.317321    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:51:01.321891 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.321861    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67cfbdf9c6-hs8j6"]
Sep 16 18:51:01.414214 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:01.414174    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" path="/var/lib/kubelet/pods/bbca215f-0c1d-47eb-a5b1-514ed6332a1b/volumes"
Sep 16 18:51:45.325946 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:45.325913    2581 scope.go:117] "RemoveContainer" containerID="c440d7efad25c4c16ecc3635b93ae9bd1823fa56843c3ccb5c2d589b980b9686"
Sep 16 18:51:55.258983 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.258952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-72gdg"]
Sep 16 18:51:55.259370 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.259352    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" containerName="console"
Sep 16 18:51:55.259413 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.259371    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" containerName="console"
Sep 16 18:51:55.259451 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.259429    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbca215f-0c1d-47eb-a5b1-514ed6332a1b" containerName="console"
Sep 16 18:51:55.263770 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.263751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.266203 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.266177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:51:55.266292 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.266259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-p898w\""
Sep 16 18:51:55.275028 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.275004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-72gdg"]
Sep 16 18:51:55.280095 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.280073    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8wjj"]
Sep 16 18:51:55.283182 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.283167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.285598 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.285581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:51:55.285766 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.285749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bt9qn\""
Sep 16 18:51:55.289355 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.289336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:51:55.300083 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.300061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8wjj"]
Sep 16 18:51:55.340936 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.340904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.341072 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.340950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmb8l\" (UniqueName: \"kubernetes.io/projected/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-kube-api-access-xmb8l\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.341072 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.340976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.341072 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.341012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8szt\" (UniqueName: \"kubernetes.io/projected/094141b9-07c2-4d85-af6f-d49fc847b434-kube-api-access-v8szt\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.341207 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.341083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.442122 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.442089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmb8l\" (UniqueName: \"kubernetes.io/projected/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-kube-api-access-xmb8l\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.442122 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.442123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.442142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8szt\" (UniqueName: \"kubernetes.io/projected/094141b9-07c2-4d85-af6f-d49fc847b434-kube-api-access-v8szt\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.442191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.442241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:51:55.442248    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:51:55.442308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs podName:e4fe785e-03d3-4530-af47-f8bdd6d8ab68 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:55.942288784 +0000 UTC m=+491.004628028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs") pod "model-serving-api-7b954ffc95-72gdg" (UID: "e4fe785e-03d3-4530-af47-f8bdd6d8ab68") : secret "model-serving-api-tls" not found
Sep 16 18:51:55.442338 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:51:55.442327    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:51:55.442607 ip-10-0-130-142 kubenswrapper[2581]: E0916 18:51:55.442375    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert podName:094141b9-07c2-4d85-af6f-d49fc847b434 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:55.942363406 +0000 UTC m=+491.004702635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert") pod "odh-model-controller-7b966d7984-w8wjj" (UID: "094141b9-07c2-4d85-af6f-d49fc847b434") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:51:55.444587 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.444565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-metrics-certs\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.452532 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.452504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmb8l\" (UniqueName: \"kubernetes.io/projected/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-kube-api-access-xmb8l\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.452912 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.452891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8szt\" (UniqueName: \"kubernetes.io/projected/094141b9-07c2-4d85-af6f-d49fc847b434-kube-api-access-v8szt\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.946920 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.946877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:55.947112 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.946961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.949356 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.949329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fe785e-03d3-4530-af47-f8bdd6d8ab68-tls-certs\") pod \"model-serving-api-7b954ffc95-72gdg\" (UID: \"e4fe785e-03d3-4530-af47-f8bdd6d8ab68\") " pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:55.949470 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:55.949329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094141b9-07c2-4d85-af6f-d49fc847b434-cert\") pod \"odh-model-controller-7b966d7984-w8wjj\" (UID: \"094141b9-07c2-4d85-af6f-d49fc847b434\") " pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:56.173762 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.173720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:51:56.192998 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.192973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:51:56.449408 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.449382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-72gdg"]
Sep 16 18:51:56.451108 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:51:56.451079    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fe785e_03d3_4530_af47_f8bdd6d8ab68.slice/crio-c7ab1a78c5c969b2e3559d99544699155a9f7da9a9958cafb0d501a8876a2c70 WatchSource:0}: Error finding container c7ab1a78c5c969b2e3559d99544699155a9f7da9a9958cafb0d501a8876a2c70: Status 404 returned error can't find the container with id c7ab1a78c5c969b2e3559d99544699155a9f7da9a9958cafb0d501a8876a2c70
Sep 16 18:51:56.491933 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.491909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-w8wjj"]
Sep 16 18:51:56.494164 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:51:56.494130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094141b9_07c2_4d85_af6f_d49fc847b434.slice/crio-46f0085d476e5bfd7c7d139fac92f1eeb2b3b0d6a3c0b33ab6a94b0c7e86c0fa WatchSource:0}: Error finding container 46f0085d476e5bfd7c7d139fac92f1eeb2b3b0d6a3c0b33ab6a94b0c7e86c0fa: Status 404 returned error can't find the container with id 46f0085d476e5bfd7c7d139fac92f1eeb2b3b0d6a3c0b33ab6a94b0c7e86c0fa
Sep 16 18:51:56.523773 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.523739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-w8wjj" event={"ID":"094141b9-07c2-4d85-af6f-d49fc847b434","Type":"ContainerStarted","Data":"46f0085d476e5bfd7c7d139fac92f1eeb2b3b0d6a3c0b33ab6a94b0c7e86c0fa"}
Sep 16 18:51:56.524860 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:51:56.524838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-72gdg" event={"ID":"e4fe785e-03d3-4530-af47-f8bdd6d8ab68","Type":"ContainerStarted","Data":"c7ab1a78c5c969b2e3559d99544699155a9f7da9a9958cafb0d501a8876a2c70"}
Sep 16 18:52:00.547105 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.547048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-72gdg" event={"ID":"e4fe785e-03d3-4530-af47-f8bdd6d8ab68","Type":"ContainerStarted","Data":"cc2f745f17afe7cb5a46210bed9ad5b1645b36c3e1f7d466d103030b5af18845"}
Sep 16 18:52:00.547573 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.547123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:52:00.548439 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.548417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-w8wjj" event={"ID":"094141b9-07c2-4d85-af6f-d49fc847b434","Type":"ContainerStarted","Data":"a0c7424a873fefa9d69b0cfe2190cbde578243796a534eaae90bd910034d6d65"}
Sep 16 18:52:00.548561 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.548545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:52:00.566102 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.566049    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-72gdg" podStartSLOduration=2.024147298 podStartE2EDuration="5.566034996s" totalImagesPullingTime="3.541887698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:00.564056551 +0000 UTC m=+495.626395814" watchObservedRunningTime="2026-09-16 18:52:00.566034996 +0000 UTC m=+495.628374281"
Sep 16 18:52:00.583694 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:00.583629    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-w8wjj" podStartSLOduration=2.037865507 podStartE2EDuration="5.583612745s" totalImagesPullingTime="3.545747238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:00.582706474 +0000 UTC m=+495.645045725" watchObservedRunningTime="2026-09-16 18:52:00.583612745 +0000 UTC m=+495.645951997"
Sep 16 18:52:11.555436 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:11.555404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-w8wjj"
Sep 16 18:52:11.557214 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:52:11.557194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-72gdg"
Sep 16 18:54:05.943034 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:05.942996    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"]
Sep 16 18:54:05.946317 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:05.946300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:05.948671 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:05.948644    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:54:05.948781 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:05.948749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 16 18:54:05.953574 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:05.953376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"]
Sep 16 18:54:06.006059 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.006024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-data\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.006239 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.006069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.006239 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.006111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfdt\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-kube-api-access-wkfdt\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.106467 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.106433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-data\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.106619 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.106477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.106619 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.106514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfdt\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-kube-api-access-wkfdt\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.106824 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.106802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-data\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.108884 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.108867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.114591 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.114570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfdt\" (UniqueName: \"kubernetes.io/projected/caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2-kube-api-access-wkfdt\") pod \"seaweedfs-tls-custom-8b44d767d-dx7gg\" (UID: \"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.255531 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.255458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"
Sep 16 18:54:06.522029 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:06.521989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-dx7gg"]
Sep 16 18:54:06.523937 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:54:06.523909    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaec21e_ce8e_4e38_bb1b_2c0fd8a55ee2.slice/crio-61b43f52a3694a5f6ff77299e8248aad7326f0c1c5308688be08aaa83ee5a822 WatchSource:0}: Error finding container 61b43f52a3694a5f6ff77299e8248aad7326f0c1c5308688be08aaa83ee5a822: Status 404 returned error can't find the container with id 61b43f52a3694a5f6ff77299e8248aad7326f0c1c5308688be08aaa83ee5a822
Sep 16 18:54:07.077203 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:07.077106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg" event={"ID":"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2","Type":"ContainerStarted","Data":"ce08295e3d44747ad6e6dc6e26c943241f99799b96347efa7921d242d8ac69f0"}
Sep 16 18:54:07.077203 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:07.077170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg" event={"ID":"caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2","Type":"ContainerStarted","Data":"61b43f52a3694a5f6ff77299e8248aad7326f0c1c5308688be08aaa83ee5a822"}
Sep 16 18:54:07.093289 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:07.093243    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-dx7gg" podStartSLOduration=1.793937164 podStartE2EDuration="2.09322931s" totalImagesPullingTime="299.292146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:07.091764081 +0000 UTC m=+622.154103330" watchObservedRunningTime="2026-09-16 18:54:07.09322931 +0000 UTC m=+622.155568559"
Sep 16 18:54:19.573797 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.573762    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"]
Sep 16 18:54:19.576885 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.576869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.578852 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.578826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 16 18:54:19.578929 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.578849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:54:19.584705 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.584682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"]
Sep 16 18:54:19.722626 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.722587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee67e50f-0359-4802-be36-9316f137c7a7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.722626 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.722628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7x9\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-kube-api-access-4z7x9\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.722837 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.722664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.824038 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.823964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee67e50f-0359-4802-be36-9316f137c7a7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.824038 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.823997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7x9\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-kube-api-access-4z7x9\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.824038 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.824019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.824369 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.824345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee67e50f-0359-4802-be36-9316f137c7a7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.826367 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.826347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.831261 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.831238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7x9\" (UniqueName: \"kubernetes.io/projected/ee67e50f-0359-4802-be36-9316f137c7a7-kube-api-access-4z7x9\") pod \"seaweedfs-tls-serving-67b7477c6d-t5xht\" (UID: \"ee67e50f-0359-4802-be36-9316f137c7a7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:19.887085 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:19.887048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"
Sep 16 18:54:20.154646 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:20.154612    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t5xht"]
Sep 16 18:54:20.157472 ip-10-0-130-142 kubenswrapper[2581]: W0916 18:54:20.157443    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee67e50f_0359_4802_be36_9316f137c7a7.slice/crio-9bb841b429e354870f52e04735bc63e9b099372367f31a5382b6663747710842 WatchSource:0}: Error finding container 9bb841b429e354870f52e04735bc63e9b099372367f31a5382b6663747710842: Status 404 returned error can't find the container with id 9bb841b429e354870f52e04735bc63e9b099372367f31a5382b6663747710842
Sep 16 18:54:20.158749 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:20.158730    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:54:21.139850 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:21.139813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht" event={"ID":"ee67e50f-0359-4802-be36-9316f137c7a7","Type":"ContainerStarted","Data":"606af4bc67ae30dd8a0589a29496e38e4c32b38b60b96b7d54abfeb8c647d5ef"}
Sep 16 18:54:21.140238 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:21.139864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht" event={"ID":"ee67e50f-0359-4802-be36-9316f137c7a7","Type":"ContainerStarted","Data":"9bb841b429e354870f52e04735bc63e9b099372367f31a5382b6663747710842"}
Sep 16 18:54:21.155560 ip-10-0-130-142 kubenswrapper[2581]: I0916 18:54:21.155511    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t5xht" podStartSLOduration=1.910445059 podStartE2EDuration="2.155494949s" totalImagesPullingTime="245.04989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:21.155304242 +0000 UTC m=+636.217643499" watchObservedRunningTime="2026-09-16 18:54:21.155494949 +0000 UTC m=+636.217834200"
Sep 16 19:00:19.077812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.077728    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:00:19.081418 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.081394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.084192 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084138    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:00:19.084292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 16 19:00:19.084292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084278    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:00:19.084400 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:00:19.084598 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:00:19.084661 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.084605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:19.085076 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.085058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:00:19.092087 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.092067    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:00:19.094482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.094462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.094584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.094519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.094584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.094557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.094695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.094629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.094695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.094653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.195837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.195803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.195837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.195837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.195877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.195909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.195931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196388 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.196363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.196645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.196705 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.196663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.198565 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.198540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.203515 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.203494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.391793 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.391757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:19.667252 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.667229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:00:19.669273 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:00:19.669247    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85054be_851f_4c7c_bd3a_c7f375ef5484.slice/crio-04fdf5b6989ad952d64cd213fb371407bcbe2cd605801b06b5d9c0ef9d834b24 WatchSource:0}: Error finding container 04fdf5b6989ad952d64cd213fb371407bcbe2cd605801b06b5d9c0ef9d834b24: Status 404 returned error can't find the container with id 04fdf5b6989ad952d64cd213fb371407bcbe2cd605801b06b5d9c0ef9d834b24
Sep 16 19:00:19.671385 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.671366    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:00:19.684989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:19.684962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerStarted","Data":"04fdf5b6989ad952d64cd213fb371407bcbe2cd605801b06b5d9c0ef9d834b24"}
Sep 16 19:00:24.711301 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:24.711263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerStarted","Data":"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"}
Sep 16 19:00:28.730171 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:28.730116    2581 generic.go:356] "Generic (PLEG): container finished" podID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerID="0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f" exitCode=0
Sep 16 19:00:28.730543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:28.730196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerDied","Data":"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"}
Sep 16 19:00:50.875122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:50.875087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerStarted","Data":"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"}
Sep 16 19:00:54.897264 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:54.897226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerStarted","Data":"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"}
Sep 16 19:00:54.897662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:54.897436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:54.897662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:54.897492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:00:54.921946 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:00:54.921893    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podStartSLOduration=2.794318897 podStartE2EDuration="35.921878638s" totalImagesPullingTime="29.374100239s" totalInitContainerRuntime="3.753459502s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:54.920300655 +0000 UTC m=+1029.982639906" watchObservedRunningTime="2026-09-16 19:00:54.921878638 +0000 UTC m=+1029.984217889"
Sep 16 19:01:00.907044 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:01:00.907017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:01:20.908333 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:01:20.908289    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:30.907955 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:01:30.907913    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:40.907659 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:01:40.907583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:50.907928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:01:50.907870    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:00.907536 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:00.907497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:10.987947 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:10.987911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:02:19.198592 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:19.198559    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:02:19.199080 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:19.198804    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" containerID="cri-o://db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85" gracePeriod=30
Sep 16 19:02:19.199080 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:19.198863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kube-rbac-proxy" containerID="cri-o://09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d" gracePeriod=30
Sep 16 19:02:20.274429 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:20.274394    2581 generic.go:356] "Generic (PLEG): container finished" podID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerID="09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d" exitCode=2
Sep 16 19:02:20.274787 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:20.274467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerDied","Data":"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"}
Sep 16 19:02:20.902707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:20.902664    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:02:20.908255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:20.908226    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:23.852884 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.852854    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:02:23.912946 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.912911    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz\") pod \"d85054be-851f-4c7c-bd3a-c7f375ef5484\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") "
Sep 16 19:02:23.913136 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.912954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls\") pod \"d85054be-851f-4c7c-bd3a-c7f375ef5484\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") "
Sep 16 19:02:23.913136 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.912989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location\") pod \"d85054be-851f-4c7c-bd3a-c7f375ef5484\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") "
Sep 16 19:02:23.913136 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.913057    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config\") pod \"d85054be-851f-4c7c-bd3a-c7f375ef5484\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") "
Sep 16 19:02:23.913357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.913330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d85054be-851f-4c7c-bd3a-c7f375ef5484" (UID: "d85054be-851f-4c7c-bd3a-c7f375ef5484"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:23.913443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.913420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d85054be-851f-4c7c-bd3a-c7f375ef5484" (UID: "d85054be-851f-4c7c-bd3a-c7f375ef5484"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:23.915274 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.915247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d85054be-851f-4c7c-bd3a-c7f375ef5484" (UID: "d85054be-851f-4c7c-bd3a-c7f375ef5484"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:23.915274 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:23.915256    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz" (OuterVolumeSpecName: "kube-api-access-vvjdz") pod "d85054be-851f-4c7c-bd3a-c7f375ef5484" (UID: "d85054be-851f-4c7c-bd3a-c7f375ef5484"). InnerVolumeSpecName "kube-api-access-vvjdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:24.014488 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert\") pod \"d85054be-851f-4c7c-bd3a-c7f375ef5484\" (UID: \"d85054be-851f-4c7c-bd3a-c7f375ef5484\") "
Sep 16 19:02:24.014625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014570    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.014625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014581    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/d85054be-851f-4c7c-bd3a-c7f375ef5484-kube-api-access-vvjdz\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.014625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014591    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d85054be-851f-4c7c-bd3a-c7f375ef5484-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.014625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014600    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d85054be-851f-4c7c-bd3a-c7f375ef5484-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.014752 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.014724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d85054be-851f-4c7c-bd3a-c7f375ef5484" (UID: "d85054be-851f-4c7c-bd3a-c7f375ef5484"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:24.115698 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.115666    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d85054be-851f-4c7c-bd3a-c7f375ef5484-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.292732 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.292642    2581 generic.go:356] "Generic (PLEG): container finished" podID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerID="db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85" exitCode=0
Sep 16 19:02:24.292732 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.292710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"
Sep 16 19:02:24.292938 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.292728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerDied","Data":"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"}
Sep 16 19:02:24.292938 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.292809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d" event={"ID":"d85054be-851f-4c7c-bd3a-c7f375ef5484","Type":"ContainerDied","Data":"04fdf5b6989ad952d64cd213fb371407bcbe2cd605801b06b5d9c0ef9d834b24"}
Sep 16 19:02:24.292938 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.292836    2581 scope.go:117] "RemoveContainer" containerID="09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"
Sep 16 19:02:24.301231 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.301211    2581 scope.go:117] "RemoveContainer" containerID="db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"
Sep 16 19:02:24.308629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.308612    2581 scope.go:117] "RemoveContainer" containerID="0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"
Sep 16 19:02:24.315495 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.315471    2581 scope.go:117] "RemoveContainer" containerID="09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"
Sep 16 19:02:24.315766 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:02:24.315749    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d\": container with ID starting with 09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d not found: ID does not exist" containerID="09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"
Sep 16 19:02:24.315827 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.315773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d"} err="failed to get container status \"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d\": rpc error: code = NotFound desc = could not find container \"09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d\": container with ID starting with 09af2ed691bf277ea5f65066b9a3539360f2cc8f72af3478d513dfed61745b1d not found: ID does not exist"
Sep 16 19:02:24.315827 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.315788    2581 scope.go:117] "RemoveContainer" containerID="db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"
Sep 16 19:02:24.316098 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:02:24.316077    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85\": container with ID starting with db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85 not found: ID does not exist" containerID="db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"
Sep 16 19:02:24.316191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.316104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85"} err="failed to get container status \"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85\": rpc error: code = NotFound desc = could not find container \"db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85\": container with ID starting with db6fe47d0538eebbe76b70bf5ca2c7f3e6ccaac96e5178d6f9a9627d99348f85 not found: ID does not exist"
Sep 16 19:02:24.316191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.316120    2581 scope.go:117] "RemoveContainer" containerID="0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"
Sep 16 19:02:24.316599 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:02:24.316573    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f\": container with ID starting with 0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f not found: ID does not exist" containerID="0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"
Sep 16 19:02:24.316693 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.316607    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f"} err="failed to get container status \"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f\": rpc error: code = NotFound desc = could not find container \"0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f\": container with ID starting with 0891f54bf8565af5e474cf56db979e88260f525fd370b294dcb02380984fea2f not found: ID does not exist"
Sep 16 19:02:24.317834 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.317815    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:02:24.327243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.327219    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d"]
Sep 16 19:02:24.373875 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.373844    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:02:24.374239 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374220    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kube-rbac-proxy"
Sep 16 19:02:24.374239 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374238    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kube-rbac-proxy"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374250    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="storage-initializer"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="storage-initializer"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374273    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374278    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374327    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kserve-container"
Sep 16 19:02:24.374332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.374333    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" containerName="kube-rbac-proxy"
Sep 16 19:02:24.379279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.379251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.382041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.381798    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:02:24.382041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.381844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:02:24.382041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.381862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:02:24.382351 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.382132    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 16 19:02:24.382351 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.382139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:02:24.382351 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.382225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:02:24.382503 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.382429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:02:24.392565 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.392541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:02:24.419353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.419328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.419473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.419357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.419473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.419396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.419473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.419436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.419644 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.419542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwmtm\" (UniqueName: \"kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520222 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwmtm\" (UniqueName: \"kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520556 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:02:24.520430    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:02:24.520556 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:02:24.520511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls podName:56c37014-7db5-4c55-950e-baa7dd7eb123 nodeName:}" failed. No retries permitted until 2026-09-16 19:02:25.020489872 +0000 UTC m=+1120.082829103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:02:24.520699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520908 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.520961 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.520944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:24.532265 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:24.532239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwmtm\" (UniqueName: \"kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:25.024388 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:25.024346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:25.026728 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:25.026706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:25.291839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:25.291755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:25.414753 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:25.414723    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85054be-851f-4c7c-bd3a-c7f375ef5484" path="/var/lib/kubelet/pods/d85054be-851f-4c7c-bd3a-c7f375ef5484/volumes"
Sep 16 19:02:25.573474 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:25.573405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:02:25.576389 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:02:25.576358    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c37014_7db5_4c55_950e_baa7dd7eb123.slice/crio-e42c8d45cb7982fe4c1f5f3a814129cdad3c70a393f911ef9995863faff0543e WatchSource:0}: Error finding container e42c8d45cb7982fe4c1f5f3a814129cdad3c70a393f911ef9995863faff0543e: Status 404 returned error can't find the container with id e42c8d45cb7982fe4c1f5f3a814129cdad3c70a393f911ef9995863faff0543e
Sep 16 19:02:26.305085 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:26.305051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerStarted","Data":"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"}
Sep 16 19:02:26.305520 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:26.305109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerStarted","Data":"e42c8d45cb7982fe4c1f5f3a814129cdad3c70a393f911ef9995863faff0543e"}
Sep 16 19:02:30.331285 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:30.331253    2581 generic.go:356] "Generic (PLEG): container finished" podID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerID="cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681" exitCode=0
Sep 16 19:02:30.331654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:30.331328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerDied","Data":"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"}
Sep 16 19:02:31.338080 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:31.338048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerStarted","Data":"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"}
Sep 16 19:02:31.338489 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:31.338109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerStarted","Data":"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"}
Sep 16 19:02:31.338489 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:31.338368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:31.368559 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:31.368513    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podStartSLOduration=3.437587048 podStartE2EDuration="7.368498101s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.930911053s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:31.367212078 +0000 UTC m=+1126.429551324" watchObservedRunningTime="2026-09-16 19:02:31.368498101 +0000 UTC m=+1126.430837350"
Sep 16 19:02:32.345254 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:32.345221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:02:38.354427 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:02:38.354395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:03:01.187159 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.187108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:03:01.192732 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.192707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.195551 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.195522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 16 19:03:01.195663 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.195545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:03:01.205367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.205344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:03:01.329100 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.329066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.329287 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.329114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.329287 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.329176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.329287 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.329219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4zf\" (UniqueName: \"kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.329287 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.329250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.429868 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.429831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.429899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.429941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4zf\" (UniqueName: \"kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.429982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.430020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.430340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.430651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.430739 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.430653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.432519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.432497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.444586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.444508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4zf\" (UniqueName: \"kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-hvzvj\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.502510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.502465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:03:01.809466 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:01.809349    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:03:01.812145 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:03:01.812118    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode493266a_885b_4b4c_a754_1b5f131e6d6c.slice/crio-228179bd49247cbb18a12279f1c8dc7cc3d6bfbdbc094ad5010933430aeab98b WatchSource:0}: Error finding container 228179bd49247cbb18a12279f1c8dc7cc3d6bfbdbc094ad5010933430aeab98b: Status 404 returned error can't find the container with id 228179bd49247cbb18a12279f1c8dc7cc3d6bfbdbc094ad5010933430aeab98b
Sep 16 19:03:02.475719 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:02.475681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerStarted","Data":"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"}
Sep 16 19:03:02.476175 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:02.475747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerStarted","Data":"228179bd49247cbb18a12279f1c8dc7cc3d6bfbdbc094ad5010933430aeab98b"}
Sep 16 19:03:08.356164 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:08.356065    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:15.534425 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:15.534390    2581 generic.go:356] "Generic (PLEG): container finished" podID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerID="9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08" exitCode=0
Sep 16 19:03:15.534765 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:15.534462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerDied","Data":"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"}
Sep 16 19:03:18.355102 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:18.355054    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:28.355794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:28.355744    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:38.355933 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:38.355888    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:48.390755 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:48.390697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:03:54.364464 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:54.364420    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:03:54.365006 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:54.364715    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" containerID="cri-o://60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1" gracePeriod=30
Sep 16 19:03:54.365006 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:54.364766    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kube-rbac-proxy" containerID="cri-o://6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b" gracePeriod=30
Sep 16 19:03:54.739244 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:54.739144    2581 generic.go:356] "Generic (PLEG): container finished" podID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerID="6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b" exitCode=2
Sep 16 19:03:54.739244 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:54.739213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerDied","Data":"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"}
Sep 16 19:03:58.350091 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:58.350037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 19:03:58.355554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:58.355518    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:59.416554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.416527    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:03:59.540850 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.540751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwmtm\" (UniqueName: \"kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm\") pod \"56c37014-7db5-4c55-950e-baa7dd7eb123\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") "
Sep 16 19:03:59.541114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.540856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") pod \"56c37014-7db5-4c55-950e-baa7dd7eb123\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") "
Sep 16 19:03:59.541114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert\") pod \"56c37014-7db5-4c55-950e-baa7dd7eb123\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") "
Sep 16 19:03:59.541114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location\") pod \"56c37014-7db5-4c55-950e-baa7dd7eb123\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") "
Sep 16 19:03:59.541114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config\") pod \"56c37014-7db5-4c55-950e-baa7dd7eb123\" (UID: \"56c37014-7db5-4c55-950e-baa7dd7eb123\") "
Sep 16 19:03:59.541830 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56c37014-7db5-4c55-950e-baa7dd7eb123" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:59.541830 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56c37014-7db5-4c55-950e-baa7dd7eb123" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:59.541830 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.541805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56c37014-7db5-4c55-950e-baa7dd7eb123" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:59.543614 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.543589    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56c37014-7db5-4c55-950e-baa7dd7eb123" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:59.543614 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.543602    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm" (OuterVolumeSpecName: "kube-api-access-wwmtm") pod "56c37014-7db5-4c55-950e-baa7dd7eb123" (UID: "56c37014-7db5-4c55-950e-baa7dd7eb123"). InnerVolumeSpecName "kube-api-access-wwmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:59.642168 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.642114    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c37014-7db5-4c55-950e-baa7dd7eb123-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:03:59.642168 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.642159    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:03:59.642168 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.642170    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c37014-7db5-4c55-950e-baa7dd7eb123-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:03:59.642462 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.642185    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:03:59.642462 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.642197    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwmtm\" (UniqueName: \"kubernetes.io/projected/56c37014-7db5-4c55-950e-baa7dd7eb123-kube-api-access-wwmtm\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:03:59.762507 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.762466    2581 generic.go:356] "Generic (PLEG): container finished" podID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerID="60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1" exitCode=0
Sep 16 19:03:59.762674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.762519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"
Sep 16 19:03:59.762674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.762549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerDied","Data":"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"}
Sep 16 19:03:59.762674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.762617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8" event={"ID":"56c37014-7db5-4c55-950e-baa7dd7eb123","Type":"ContainerDied","Data":"e42c8d45cb7982fe4c1f5f3a814129cdad3c70a393f911ef9995863faff0543e"}
Sep 16 19:03:59.762674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.762644    2581 scope.go:117] "RemoveContainer" containerID="6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"
Sep 16 19:03:59.771693 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.771673    2581 scope.go:117] "RemoveContainer" containerID="60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"
Sep 16 19:03:59.779587 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.779565    2581 scope.go:117] "RemoveContainer" containerID="cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"
Sep 16 19:03:59.786317 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.786291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:03:59.788582 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.788539    2581 scope.go:117] "RemoveContainer" containerID="6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"
Sep 16 19:03:59.788852 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:03:59.788828    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b\": container with ID starting with 6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b not found: ID does not exist" containerID="6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"
Sep 16 19:03:59.788949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.788863    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b"} err="failed to get container status \"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b\": rpc error: code = NotFound desc = could not find container \"6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b\": container with ID starting with 6999c0f8450bca0c96c06ae6a79bb37195ccd60f0c197c58b2240ce0ba5fe39b not found: ID does not exist"
Sep 16 19:03:59.788949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.788887    2581 scope.go:117] "RemoveContainer" containerID="60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"
Sep 16 19:03:59.789206 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:03:59.789183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1\": container with ID starting with 60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1 not found: ID does not exist" containerID="60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"
Sep 16 19:03:59.789296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.789212    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1"} err="failed to get container status \"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1\": rpc error: code = NotFound desc = could not find container \"60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1\": container with ID starting with 60af0da7367cf52c24fc62e3fae9592d0c944eee8abace19a26163a21473efd1 not found: ID does not exist"
Sep 16 19:03:59.789296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.789227    2581 scope.go:117] "RemoveContainer" containerID="cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"
Sep 16 19:03:59.789526 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:03:59.789506    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681\": container with ID starting with cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681 not found: ID does not exist" containerID="cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"
Sep 16 19:03:59.789633 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.789528    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681"} err="failed to get container status \"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681\": rpc error: code = NotFound desc = could not find container \"cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681\": container with ID starting with cd142484a068eeee94e97675edb196558a911263d2184c009765eff505add681 not found: ID does not exist"
Sep 16 19:03:59.790199 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:03:59.790094    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8"]
Sep 16 19:04:01.414396 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:04:01.414365    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" path="/var/lib/kubelet/pods/56c37014-7db5-4c55-950e-baa7dd7eb123/volumes"
Sep 16 19:05:19.157018 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:19.156978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerStarted","Data":"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"}
Sep 16 19:05:19.157472 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:19.157049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerStarted","Data":"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"}
Sep 16 19:05:19.157472 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:19.157195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:05:19.181936 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:19.181881    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" podStartSLOduration=1.8283375670000002 podStartE2EDuration="2m18.181866049s" totalImagesPullingTime="2m2.723952886s" totalInitContainerRuntime="13.629575596s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:19.180825658 +0000 UTC m=+1294.243164913" watchObservedRunningTime="2026-09-16 19:05:19.181866049 +0000 UTC m=+1294.244205300"
Sep 16 19:05:20.162619 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:20.162591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:05:26.171478 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:05:26.171447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:06:55.415117 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:06:55.415086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:07:06.452373 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:06.452320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:07:07.480715 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:07.480682    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:07:07.481111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:07.480923    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kserve-container" containerID="cri-o://531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7" gracePeriod=30
Sep 16 19:07:07.481111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:07.480989    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kube-rbac-proxy" containerID="cri-o://73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6" gracePeriod=30
Sep 16 19:07:07.635933 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:07.635898    2581 generic.go:356] "Generic (PLEG): container finished" podID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerID="73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6" exitCode=2
Sep 16 19:07:07.636094 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:07.635970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerDied","Data":"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"}
Sep 16 19:07:11.167342 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:11.167298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:07:12.527442 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.527417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:07:12.658810 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.658734    2581 generic.go:356] "Generic (PLEG): container finished" podID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerID="531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7" exitCode=0
Sep 16 19:07:12.658810 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.658794    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"
Sep 16 19:07:12.659067 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.658815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerDied","Data":"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"}
Sep 16 19:07:12.659067 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.658887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj" event={"ID":"e493266a-885b-4b4c-a754-1b5f131e6d6c","Type":"ContainerDied","Data":"228179bd49247cbb18a12279f1c8dc7cc3d6bfbdbc094ad5010933430aeab98b"}
Sep 16 19:07:12.659067 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.658926    2581 scope.go:117] "RemoveContainer" containerID="73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"
Sep 16 19:07:12.666922 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.666903    2581 scope.go:117] "RemoveContainer" containerID="531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"
Sep 16 19:07:12.668995 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.668978    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location\") pod \"e493266a-885b-4b4c-a754-1b5f131e6d6c\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") "
Sep 16 19:07:12.669069 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config\") pod \"e493266a-885b-4b4c-a754-1b5f131e6d6c\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") "
Sep 16 19:07:12.669122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp4zf\" (UniqueName: \"kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf\") pod \"e493266a-885b-4b4c-a754-1b5f131e6d6c\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") "
Sep 16 19:07:12.669486 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e493266a-885b-4b4c-a754-1b5f131e6d6c" (UID: "e493266a-885b-4b4c-a754-1b5f131e6d6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:12.669641 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls\") pod \"e493266a-885b-4b4c-a754-1b5f131e6d6c\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") "
Sep 16 19:07:12.669709 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert\") pod \"e493266a-885b-4b4c-a754-1b5f131e6d6c\" (UID: \"e493266a-885b-4b4c-a754-1b5f131e6d6c\") "
Sep 16 19:07:12.669975 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.669927    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:07:12.670228 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.670192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e493266a-885b-4b4c-a754-1b5f131e6d6c" (UID: "e493266a-885b-4b4c-a754-1b5f131e6d6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:12.671317 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.671295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf" (OuterVolumeSpecName: "kube-api-access-wp4zf") pod "e493266a-885b-4b4c-a754-1b5f131e6d6c" (UID: "e493266a-885b-4b4c-a754-1b5f131e6d6c"). InnerVolumeSpecName "kube-api-access-wp4zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:12.671396 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.671320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e493266a-885b-4b4c-a754-1b5f131e6d6c" (UID: "e493266a-885b-4b4c-a754-1b5f131e6d6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:12.672951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.672928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e493266a-885b-4b4c-a754-1b5f131e6d6c" (UID: "e493266a-885b-4b4c-a754-1b5f131e6d6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:12.675036 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.675009    2581 scope.go:117] "RemoveContainer" containerID="9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"
Sep 16 19:07:12.682665 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.682647    2581 scope.go:117] "RemoveContainer" containerID="73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"
Sep 16 19:07:12.682915 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:07:12.682892    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6\": container with ID starting with 73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6 not found: ID does not exist" containerID="73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"
Sep 16 19:07:12.682966 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.682917    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6"} err="failed to get container status \"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6\": rpc error: code = NotFound desc = could not find container \"73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6\": container with ID starting with 73ecd90c58eb8c7bc3a048ba0121b234983c0471649132caa55a604a71dcc4e6 not found: ID does not exist"
Sep 16 19:07:12.682966 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.682935    2581 scope.go:117] "RemoveContainer" containerID="531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"
Sep 16 19:07:12.683239 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:07:12.683221    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7\": container with ID starting with 531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7 not found: ID does not exist" containerID="531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"
Sep 16 19:07:12.683295 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.683248    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7"} err="failed to get container status \"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7\": rpc error: code = NotFound desc = could not find container \"531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7\": container with ID starting with 531387d9665ce11102c421a27b07d1ac0e501d8bf56e50f6bb7841ee40a0faf7 not found: ID does not exist"
Sep 16 19:07:12.683295 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.683264    2581 scope.go:117] "RemoveContainer" containerID="9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"
Sep 16 19:07:12.683533 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:07:12.683516    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08\": container with ID starting with 9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08 not found: ID does not exist" containerID="9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"
Sep 16 19:07:12.683591 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.683537    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08"} err="failed to get container status \"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08\": rpc error: code = NotFound desc = could not find container \"9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08\": container with ID starting with 9705d7643fda80154c9be8c15923e8d82d35103dd55c1769db262978fa1e6a08 not found: ID does not exist"
Sep 16 19:07:12.687138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:07:12.687512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container"
Sep 16 19:07:12.687585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container"
Sep 16 19:07:12.687585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687533    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687542    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687560    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687569    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687587    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kserve-container"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687595    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kserve-container"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687607    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="storage-initializer"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687616    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="storage-initializer"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687629    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="storage-initializer"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687637    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="storage-initializer"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kserve-container"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687746    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c37014-7db5-4c55-950e-baa7dd7eb123" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687757    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kube-rbac-proxy"
Sep 16 19:07:12.687832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.687768    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" containerName="kserve-container"
Sep 16 19:07:12.691362 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.691346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.693514 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.693497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 16 19:07:12.693600 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.693561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 16 19:07:12.700927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.700905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:07:12.770851 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.770815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.771035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.770867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnjz\" (UniqueName: \"kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.771035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.770928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.771035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.770986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.771229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.771049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.771229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.771112    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e493266a-885b-4b4c-a754-1b5f131e6d6c-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:07:12.771229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.771130    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e493266a-885b-4b4c-a754-1b5f131e6d6c-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:07:12.771229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.771144    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4zf\" (UniqueName: \"kubernetes.io/projected/e493266a-885b-4b4c-a754-1b5f131e6d6c-kube-api-access-wp4zf\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:07:12.771229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.771185    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e493266a-885b-4b4c-a754-1b5f131e6d6c-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:07:12.872256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnjz\" (UniqueName: \"kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.872256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.872445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.872445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.872445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.872559 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:07:12.872468    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:07:12.872559 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:07:12.872543    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls podName:af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f nodeName:}" failed. No retries permitted until 2026-09-16 19:07:13.372523875 +0000 UTC m=+1408.434863109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:07:12.872731 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.872711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.873087 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.873063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.873216 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.873087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.881273 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.881253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnjz\" (UniqueName: \"kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:12.982041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.982005    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:07:12.983558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:12.983535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj"]
Sep 16 19:07:13.376132 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:13.376101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:13.378459 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:13.378441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:13.414715 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:13.414689    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e493266a-885b-4b4c-a754-1b5f131e6d6c" path="/var/lib/kubelet/pods/e493266a-885b-4b4c-a754-1b5f131e6d6c/volumes"
Sep 16 19:07:13.601857 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:13.601820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:13.870707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:13.870682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:07:13.872933 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:07:13.872902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7ba2f7_df6e_46d3_a6ff_ebb471b7ca9f.slice/crio-f2162e17faf50f3731f35d3611a654cddf4231d0cf571690e795ac1ba379664a WatchSource:0}: Error finding container f2162e17faf50f3731f35d3611a654cddf4231d0cf571690e795ac1ba379664a: Status 404 returned error can't find the container with id f2162e17faf50f3731f35d3611a654cddf4231d0cf571690e795ac1ba379664a
Sep 16 19:07:14.672868 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:14.672827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerStarted","Data":"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"}
Sep 16 19:07:14.673284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:14.672893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerStarted","Data":"f2162e17faf50f3731f35d3611a654cddf4231d0cf571690e795ac1ba379664a"}
Sep 16 19:07:43.800324 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:43.800287    2581 generic.go:356] "Generic (PLEG): container finished" podID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerID="5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c" exitCode=0
Sep 16 19:07:43.800324 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:43.800328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerDied","Data":"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"}
Sep 16 19:07:44.809016 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:44.808980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerStarted","Data":"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"}
Sep 16 19:07:44.809434 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:44.809044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerStarted","Data":"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"}
Sep 16 19:07:44.809434 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:44.809296    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:44.829430 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:44.829378    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" podStartSLOduration=3.474143454 podStartE2EDuration="32.829362552s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.355219098s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:44.82811849 +0000 UTC m=+1439.890457741" watchObservedRunningTime="2026-09-16 19:07:44.829362552 +0000 UTC m=+1439.891701802"
Sep 16 19:07:45.814249 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:45.814214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:07:51.824287 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:07:51.824257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:09:13.414860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:13.414786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:09:24.450425 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:24.450358    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:09:29.389370 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:29.389331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:09:29.389776 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:29.389611    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kserve-container" containerID="cri-o://79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb" gracePeriod=30
Sep 16 19:09:29.389776 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:29.389634    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kube-rbac-proxy" containerID="cri-o://edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374" gracePeriod=30
Sep 16 19:09:30.274945 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.274906    2581 generic.go:356] "Generic (PLEG): container finished" podID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerID="edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374" exitCode=2
Sep 16 19:09:30.275124 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.274961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerDied","Data":"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"}
Sep 16 19:09:30.729178 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.729139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:09:30.869530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdnjz\" (UniqueName: \"kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz\") pod \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") "
Sep 16 19:09:30.869744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert\") pod \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") "
Sep 16 19:09:30.869744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config\") pod \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") "
Sep 16 19:09:30.869744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") pod \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") "
Sep 16 19:09:30.869744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location\") pod \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\" (UID: \"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f\") "
Sep 16 19:09:30.869974 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.869906    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:30.870039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.870012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:30.871768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.871736    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz" (OuterVolumeSpecName: "kube-api-access-wdnjz") pod "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f"). InnerVolumeSpecName "kube-api-access-wdnjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:30.871863 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.871827    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:30.873984 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.873960    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" (UID: "af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:30.971293 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.971263    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.971293 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.971288    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.971293 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.971298    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnjz\" (UniqueName: \"kubernetes.io/projected/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-api-access-wdnjz\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.971512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.971308    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.971512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:30.971316    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:09:31.281356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.281266    2581 generic.go:356] "Generic (PLEG): container finished" podID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerID="79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb" exitCode=0
Sep 16 19:09:31.281356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.281335    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"
Sep 16 19:09:31.281356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.281340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerDied","Data":"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"}
Sep 16 19:09:31.281572 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.281400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx" event={"ID":"af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f","Type":"ContainerDied","Data":"f2162e17faf50f3731f35d3611a654cddf4231d0cf571690e795ac1ba379664a"}
Sep 16 19:09:31.281572 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.281430    2581 scope.go:117] "RemoveContainer" containerID="edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"
Sep 16 19:09:31.289746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.289725    2581 scope.go:117] "RemoveContainer" containerID="79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"
Sep 16 19:09:31.297021 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.297002    2581 scope.go:117] "RemoveContainer" containerID="5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"
Sep 16 19:09:31.302560 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.302537    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:09:31.304748 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.304728    2581 scope.go:117] "RemoveContainer" containerID="edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"
Sep 16 19:09:31.305072 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:09:31.305053    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374\": container with ID starting with edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374 not found: ID does not exist" containerID="edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"
Sep 16 19:09:31.305135 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.305081    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374"} err="failed to get container status \"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374\": rpc error: code = NotFound desc = could not find container \"edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374\": container with ID starting with edb2d5539a40f7ead6ee000b6c685346417e40b707e951a06fefb70834a28374 not found: ID does not exist"
Sep 16 19:09:31.305135 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.305099    2581 scope.go:117] "RemoveContainer" containerID="79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"
Sep 16 19:09:31.305386 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:09:31.305366    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb\": container with ID starting with 79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb not found: ID does not exist" containerID="79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"
Sep 16 19:09:31.305428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.305390    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb"} err="failed to get container status \"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb\": rpc error: code = NotFound desc = could not find container \"79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb\": container with ID starting with 79639a3665f9462988d6c9e888e677cdcbb3af823a73ca0dc6fcd1c600f095bb not found: ID does not exist"
Sep 16 19:09:31.305428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.305406    2581 scope.go:117] "RemoveContainer" containerID="5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"
Sep 16 19:09:31.305663 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:09:31.305643    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c\": container with ID starting with 5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c not found: ID does not exist" containerID="5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"
Sep 16 19:09:31.305744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.305667    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c"} err="failed to get container status \"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c\": rpc error: code = NotFound desc = could not find container \"5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c\": container with ID starting with 5484a8fe56e655cc167fe23a429ebd609508ed7ba513d25c59ed19b5c2cba38c not found: ID does not exist"
Sep 16 19:09:31.306532 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.306513    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx"]
Sep 16 19:09:31.414857 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:31.414815    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" path="/var/lib/kubelet/pods/af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f/volumes"
Sep 16 19:09:34.520146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520112    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kube-rbac-proxy"
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520488    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kube-rbac-proxy"
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="storage-initializer"
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520504    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="storage-initializer"
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520518    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kserve-container"
Sep 16 19:09:34.520523 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520523    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kserve-container"
Sep 16 19:09:34.520704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520588    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kserve-container"
Sep 16 19:09:34.520704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.520597    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7ba2f7-df6e-46d3-a6ff-ebb471b7ca9f" containerName="kube-rbac-proxy"
Sep 16 19:09:34.525335 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.525316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.527767 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.527741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:34.527879 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.527861    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:09:34.527965 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.527949    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:34.528062 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.528047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:09:34.528114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.528089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 19:09:34.528174 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.528125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:34.528505 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.528491    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:34.536235 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.536210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:09:34.602069 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.602034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.602243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.602079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.602243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.602121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.602243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.602176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.602243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.602223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779tk\" (UniqueName: \"kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703042 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779tk\" (UniqueName: \"kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703601 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703740 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.703795 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.703777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.705645 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.705625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.713405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.713386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779tk\" (UniqueName: \"kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:34.840707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:34.840608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:35.114660 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:35.114610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:09:35.117410 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:09:35.117385    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba5974d_e10c_4c34_9e28_76de3e54851f.slice/crio-f9901f9f12c039dd6485f1334817e97b1319237ac8178f6d1c5046363cce7eb7 WatchSource:0}: Error finding container f9901f9f12c039dd6485f1334817e97b1319237ac8178f6d1c5046363cce7eb7: Status 404 returned error can't find the container with id f9901f9f12c039dd6485f1334817e97b1319237ac8178f6d1c5046363cce7eb7
Sep 16 19:09:35.302857 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:35.302822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerStarted","Data":"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"}
Sep 16 19:09:35.303013 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:35.302879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerStarted","Data":"f9901f9f12c039dd6485f1334817e97b1319237ac8178f6d1c5046363cce7eb7"}
Sep 16 19:09:40.325540 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:40.325459    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerID="da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0" exitCode=0
Sep 16 19:09:40.325887 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:40.325535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerDied","Data":"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"}
Sep 16 19:09:41.332060 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:41.332021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerStarted","Data":"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"}
Sep 16 19:09:41.332562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:41.332083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerStarted","Data":"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"}
Sep 16 19:09:41.332562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:41.332280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:41.350971 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:41.350919    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" podStartSLOduration=2.534167166 podStartE2EDuration="7.350902956s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.81673579s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:41.349261511 +0000 UTC m=+1556.411600761" watchObservedRunningTime="2026-09-16 19:09:41.350902956 +0000 UTC m=+1556.413242204"
Sep 16 19:09:42.337839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:42.337802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:09:48.348717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:09:48.348686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:11:10.413546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:10.413501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:11:20.738667 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:20.738629    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:11:20.739078 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:20.738864    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kserve-container" containerID="cri-o://47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4" gracePeriod=30
Sep 16 19:11:20.739078 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:20.738898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kube-rbac-proxy" containerID="cri-o://8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2" gracePeriod=30
Sep 16 19:11:21.779491 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:21.779454    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerID="8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2" exitCode=2
Sep 16 19:11:21.779875 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:21.779493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerDied","Data":"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"}
Sep 16 19:11:22.189385 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.189362    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:11:22.297454 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location\") pod \"5ba5974d-e10c-4c34-9e28-76de3e54851f\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") "
Sep 16 19:11:22.297454 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls\") pod \"5ba5974d-e10c-4c34-9e28-76de3e54851f\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") "
Sep 16 19:11:22.297687 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config\") pod \"5ba5974d-e10c-4c34-9e28-76de3e54851f\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") "
Sep 16 19:11:22.297687 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-779tk\" (UniqueName: \"kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk\") pod \"5ba5974d-e10c-4c34-9e28-76de3e54851f\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") "
Sep 16 19:11:22.297687 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert\") pod \"5ba5974d-e10c-4c34-9e28-76de3e54851f\" (UID: \"5ba5974d-e10c-4c34-9e28-76de3e54851f\") "
Sep 16 19:11:22.297991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.297964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ba5974d-e10c-4c34-9e28-76de3e54851f" (UID: "5ba5974d-e10c-4c34-9e28-76de3e54851f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:22.298096 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.298044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ba5974d-e10c-4c34-9e28-76de3e54851f" (UID: "5ba5974d-e10c-4c34-9e28-76de3e54851f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:22.299702 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.299672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ba5974d-e10c-4c34-9e28-76de3e54851f" (UID: "5ba5974d-e10c-4c34-9e28-76de3e54851f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:22.299841 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.299721    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk" (OuterVolumeSpecName: "kube-api-access-779tk") pod "5ba5974d-e10c-4c34-9e28-76de3e54851f" (UID: "5ba5974d-e10c-4c34-9e28-76de3e54851f"). InnerVolumeSpecName "kube-api-access-779tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:22.301854 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.301830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ba5974d-e10c-4c34-9e28-76de3e54851f" (UID: "5ba5974d-e10c-4c34-9e28-76de3e54851f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:22.398554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.398508    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba5974d-e10c-4c34-9e28-76de3e54851f-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:11:22.398554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.398550    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba5974d-e10c-4c34-9e28-76de3e54851f-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:11:22.398554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.398562    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:11:22.398825 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.398571    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779tk\" (UniqueName: \"kubernetes.io/projected/5ba5974d-e10c-4c34-9e28-76de3e54851f-kube-api-access-779tk\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:11:22.398825 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.398584    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba5974d-e10c-4c34-9e28-76de3e54851f-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:11:22.785477 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.785392    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerID="47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4" exitCode=0
Sep 16 19:11:22.785477 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.785446    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"
Sep 16 19:11:22.786017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.785476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerDied","Data":"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"}
Sep 16 19:11:22.786017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.785546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z" event={"ID":"5ba5974d-e10c-4c34-9e28-76de3e54851f","Type":"ContainerDied","Data":"f9901f9f12c039dd6485f1334817e97b1319237ac8178f6d1c5046363cce7eb7"}
Sep 16 19:11:22.786017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.785584    2581 scope.go:117] "RemoveContainer" containerID="8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"
Sep 16 19:11:22.794966 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.794945    2581 scope.go:117] "RemoveContainer" containerID="47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"
Sep 16 19:11:22.802223 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.802205    2581 scope.go:117] "RemoveContainer" containerID="da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"
Sep 16 19:11:22.808790 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.808768    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:11:22.809608 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.809593    2581 scope.go:117] "RemoveContainer" containerID="8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"
Sep 16 19:11:22.809886 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:11:22.809864    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2\": container with ID starting with 8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2 not found: ID does not exist" containerID="8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"
Sep 16 19:11:22.809951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.809892    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2"} err="failed to get container status \"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2\": rpc error: code = NotFound desc = could not find container \"8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2\": container with ID starting with 8d7bce3e6276158a3719b1f8c8abd8c1c567b70c615c78870be01587e86706e2 not found: ID does not exist"
Sep 16 19:11:22.809951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.809909    2581 scope.go:117] "RemoveContainer" containerID="47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"
Sep 16 19:11:22.810187 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:11:22.810141    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4\": container with ID starting with 47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4 not found: ID does not exist" containerID="47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"
Sep 16 19:11:22.810235 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.810198    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4"} err="failed to get container status \"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4\": rpc error: code = NotFound desc = could not find container \"47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4\": container with ID starting with 47ac39614970bf84e5c10dbe78d3f99794d08e2ebeedf396abfab4d310857ac4 not found: ID does not exist"
Sep 16 19:11:22.810235 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.810219    2581 scope.go:117] "RemoveContainer" containerID="da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"
Sep 16 19:11:22.810507 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:11:22.810482    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0\": container with ID starting with da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0 not found: ID does not exist" containerID="da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"
Sep 16 19:11:22.810554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.810503    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0"} err="failed to get container status \"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0\": rpc error: code = NotFound desc = could not find container \"da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0\": container with ID starting with da4f014ac2bff60ca070d8b1dcfcb13dd3fe2ddb793d75b5880b871680f782a0 not found: ID does not exist"
Sep 16 19:11:22.814281 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:22.814262    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z"]
Sep 16 19:11:23.419011 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:23.418972    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" path="/var/lib/kubelet/pods/5ba5974d-e10c-4c34-9e28-76de3e54851f/volumes"
Sep 16 19:11:25.920039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920001    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kserve-container"
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kserve-container"
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920399    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="storage-initializer"
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920405    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="storage-initializer"
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920419    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kube-rbac-proxy"
Sep 16 19:11:25.920445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kube-rbac-proxy"
Sep 16 19:11:25.920639 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920488    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kube-rbac-proxy"
Sep 16 19:11:25.920639 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.920499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba5974d-e10c-4c34-9e28-76de3e54851f" containerName="kserve-container"
Sep 16 19:11:25.925518 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.925499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:25.927937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.927909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:25.928075 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.927910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:11:25.928075 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.927957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:25.928230 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.927961    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:25.928230 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.927982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:25.928230 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.928116    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 16 19:11:25.928747 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.928731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:11:25.933236 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:25.933215    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:11:26.027371 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.027331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8j6\" (UniqueName: \"kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.027546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.027393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.027546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.027435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.027546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.027468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.027546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.027535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128622 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.128582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.128637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.128672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.128727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8j6\" (UniqueName: \"kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:11:26.128734    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.128769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.128819 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:11:26.128802    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls podName:947328c0-efce-4252-8181-b191e6c3c8e1 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:26.628780586 +0000 UTC m=+1661.691119816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:11:26.129242 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.129074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.129538 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.129519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.129616 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.129520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.137545 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.137523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8j6\" (UniqueName: \"kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.633058 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.633015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.635503 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.635469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:26.836353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:26.836318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:27.110661 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:27.110632    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:11:27.112735 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:11:27.112705    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947328c0_efce_4252_8181_b191e6c3c8e1.slice/crio-42789290adabc1e241189a8becd575b2dc25e3ad0b29c3883511d57f9ad2ea80 WatchSource:0}: Error finding container 42789290adabc1e241189a8becd575b2dc25e3ad0b29c3883511d57f9ad2ea80: Status 404 returned error can't find the container with id 42789290adabc1e241189a8becd575b2dc25e3ad0b29c3883511d57f9ad2ea80
Sep 16 19:11:27.810435 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:27.810401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerStarted","Data":"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"}
Sep 16 19:11:27.810638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:27.810457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerStarted","Data":"42789290adabc1e241189a8becd575b2dc25e3ad0b29c3883511d57f9ad2ea80"}
Sep 16 19:11:33.839173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:33.839119    2581 generic.go:356] "Generic (PLEG): container finished" podID="947328c0-efce-4252-8181-b191e6c3c8e1" containerID="ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7" exitCode=0
Sep 16 19:11:33.839530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:33.839195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerDied","Data":"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"}
Sep 16 19:11:34.845599 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:34.845564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerStarted","Data":"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"}
Sep 16 19:11:34.845975 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:34.845622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerStarted","Data":"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"}
Sep 16 19:11:34.845975 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:34.845808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:34.871664 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:34.871612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podStartSLOduration=4.024223445 podStartE2EDuration="9.871596701s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.847373256s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:34.869636535 +0000 UTC m=+1669.931975798" watchObservedRunningTime="2026-09-16 19:11:34.871596701 +0000 UTC m=+1669.933936001"
Sep 16 19:11:35.851360 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:35.851321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:35.852804 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:35.852777    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:11:36.857229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:36.857189    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:11:41.861304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:41.861270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:11:41.861926 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:41.861897    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:11:51.862956 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:11:51.862922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:12:08.156506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:08.156430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:12:08.156865 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:08.156702    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container" containerID="cri-o://39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352" gracePeriod=30
Sep 16 19:12:08.156865 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:08.156754    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kube-rbac-proxy" containerID="cri-o://8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af" gracePeriod=30
Sep 16 19:12:09.015715 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.015676    2581 generic.go:356] "Generic (PLEG): container finished" podID="947328c0-efce-4252-8181-b191e6c3c8e1" containerID="8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af" exitCode=2
Sep 16 19:12:09.015905 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.015756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerDied","Data":"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"}
Sep 16 19:12:09.807925 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.807896    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:12:09.908996 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.908902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") pod \"947328c0-efce-4252-8181-b191e6c3c8e1\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") "
Sep 16 19:12:09.908996 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.908949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr8j6\" (UniqueName: \"kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6\") pod \"947328c0-efce-4252-8181-b191e6c3c8e1\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") "
Sep 16 19:12:09.908996 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.908978    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location\") pod \"947328c0-efce-4252-8181-b191e6c3c8e1\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") "
Sep 16 19:12:09.909294 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.909011    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config\") pod \"947328c0-efce-4252-8181-b191e6c3c8e1\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") "
Sep 16 19:12:09.909294 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.909054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert\") pod \"947328c0-efce-4252-8181-b191e6c3c8e1\" (UID: \"947328c0-efce-4252-8181-b191e6c3c8e1\") "
Sep 16 19:12:09.909473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.909446    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "947328c0-efce-4252-8181-b191e6c3c8e1" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:09.909542 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.909518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "947328c0-efce-4252-8181-b191e6c3c8e1" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:09.910073 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.910048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "947328c0-efce-4252-8181-b191e6c3c8e1" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:09.911240 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.911216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6" (OuterVolumeSpecName: "kube-api-access-nr8j6") pod "947328c0-efce-4252-8181-b191e6c3c8e1" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1"). InnerVolumeSpecName "kube-api-access-nr8j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:09.911305 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:09.911229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "947328c0-efce-4252-8181-b191e6c3c8e1" (UID: "947328c0-efce-4252-8181-b191e6c3c8e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:10.010288 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.010255    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.010288 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.010284    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/947328c0-efce-4252-8181-b191e6c3c8e1-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.010288 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.010293    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8j6\" (UniqueName: \"kubernetes.io/projected/947328c0-efce-4252-8181-b191e6c3c8e1-kube-api-access-nr8j6\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.010534 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.010302    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/947328c0-efce-4252-8181-b191e6c3c8e1-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.010534 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.010311    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/947328c0-efce-4252-8181-b191e6c3c8e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:10.021777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.021746    2581 generic.go:356] "Generic (PLEG): container finished" podID="947328c0-efce-4252-8181-b191e6c3c8e1" containerID="39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352" exitCode=0
Sep 16 19:12:10.021919 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.021801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"
Sep 16 19:12:10.021919 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.021835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerDied","Data":"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"}
Sep 16 19:12:10.021919 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.021907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p" event={"ID":"947328c0-efce-4252-8181-b191e6c3c8e1","Type":"ContainerDied","Data":"42789290adabc1e241189a8becd575b2dc25e3ad0b29c3883511d57f9ad2ea80"}
Sep 16 19:12:10.022034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.021954    2581 scope.go:117] "RemoveContainer" containerID="8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"
Sep 16 19:12:10.030310 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.030291    2581 scope.go:117] "RemoveContainer" containerID="39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"
Sep 16 19:12:10.037735 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.037714    2581 scope.go:117] "RemoveContainer" containerID="ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"
Sep 16 19:12:10.045094 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045071    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:12:10.045296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045282    2581 scope.go:117] "RemoveContainer" containerID="8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"
Sep 16 19:12:10.045599 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:10.045579    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af\": container with ID starting with 8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af not found: ID does not exist" containerID="8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"
Sep 16 19:12:10.045656 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045606    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af"} err="failed to get container status \"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af\": rpc error: code = NotFound desc = could not find container \"8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af\": container with ID starting with 8315cbb1850c07297c04a7c221cfa8ea5f867f479eb5aef400196bb12bde62af not found: ID does not exist"
Sep 16 19:12:10.045656 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045626    2581 scope.go:117] "RemoveContainer" containerID="39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"
Sep 16 19:12:10.045904 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:10.045885    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352\": container with ID starting with 39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352 not found: ID does not exist" containerID="39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"
Sep 16 19:12:10.045958 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045910    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352"} err="failed to get container status \"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352\": rpc error: code = NotFound desc = could not find container \"39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352\": container with ID starting with 39709b39377d1761c8684508ee02174a0025d4219ecbb2596daaf2b5ebe9a352 not found: ID does not exist"
Sep 16 19:12:10.045958 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.045932    2581 scope.go:117] "RemoveContainer" containerID="ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"
Sep 16 19:12:10.046202 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:10.046183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7\": container with ID starting with ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7 not found: ID does not exist" containerID="ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"
Sep 16 19:12:10.046263 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.046209    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7"} err="failed to get container status \"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7\": rpc error: code = NotFound desc = could not find container \"ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7\": container with ID starting with ac689a88af5a92861ca344e8c357ea80a220a8d96c3e27cae4f21377e44a10c7 not found: ID does not exist"
Sep 16 19:12:10.052109 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:10.052087    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p"]
Sep 16 19:12:11.414669 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:11.414633    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" path="/var/lib/kubelet/pods/947328c0-efce-4252-8181-b191e6c3c8e1/volumes"
Sep 16 19:12:13.347941 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.347907    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348246    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="storage-initializer"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348257    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="storage-initializer"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348278    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kube-rbac-proxy"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348284    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kube-rbac-proxy"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348295    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348301    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348352    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kube-rbac-proxy"
Sep 16 19:12:13.348405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.348363    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="947328c0-efce-4252-8181-b191e6c3c8e1" containerName="kserve-container"
Sep 16 19:12:13.353116 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.353097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.356323 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.356305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 16 19:12:13.357254 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:13.357369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:12:13.357369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357272    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:12:13.357369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357333    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:13.357562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:12:13.357645 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.357630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:12:13.362644 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.362619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:13.439825 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.439794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.439825 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.439823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.440051 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.439886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k4v\" (UniqueName: \"kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.440051 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.439914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.440051 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.439928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.540662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.540625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.540662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.540661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.540902 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.540803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.540902 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.540843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.541003 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.540964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k4v\" (UniqueName: \"kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.541284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.541254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.541386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.541367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.541455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.541431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.543096 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.543077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.549224 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.549199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k4v\" (UniqueName: \"kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.663809 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.663728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:13.936320 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:13.936293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:13.938345 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:12:13.938320    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f74688_aed4_4ea8_a5c0_f66e4c159b1c.slice/crio-451de08ca9119b2dd8a0618043ffbf772c3a8f38163be60d15f25c6b9a428dd1 WatchSource:0}: Error finding container 451de08ca9119b2dd8a0618043ffbf772c3a8f38163be60d15f25c6b9a428dd1: Status 404 returned error can't find the container with id 451de08ca9119b2dd8a0618043ffbf772c3a8f38163be60d15f25c6b9a428dd1
Sep 16 19:12:14.042801 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:14.042765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerStarted","Data":"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"}
Sep 16 19:12:14.042956 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:14.042825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerStarted","Data":"451de08ca9119b2dd8a0618043ffbf772c3a8f38163be60d15f25c6b9a428dd1"}
Sep 16 19:12:20.070032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:20.069996    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerID="958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4" exitCode=0
Sep 16 19:12:20.070470 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:20.070069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerDied","Data":"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"}
Sep 16 19:12:21.077773 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.077741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerStarted","Data":"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"}
Sep 16 19:12:21.078207 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.077803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerStarted","Data":"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"}
Sep 16 19:12:21.078207 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.078097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:21.078297 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.078268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:21.079868 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.079842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:12:21.098683 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:21.098642    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podStartSLOduration=2.978032733 podStartE2EDuration="8.098629522s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.120596789s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:21.09768014 +0000 UTC m=+1716.160019393" watchObservedRunningTime="2026-09-16 19:12:21.098629522 +0000 UTC m=+1716.160968773"
Sep 16 19:12:22.082627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:22.082581    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:12:27.087679 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:27.087651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:27.088111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:27.088089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:12:37.089524 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:37.089491    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:48.478315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:48.478280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:48.478818 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:48.478575    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container" containerID="cri-o://608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa" gracePeriod=30
Sep 16 19:12:48.478818 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:48.478618    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kube-rbac-proxy" containerID="cri-o://b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724" gracePeriod=30
Sep 16 19:12:49.211856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.211820    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerID="b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724" exitCode=2
Sep 16 19:12:49.212022 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.211889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerDied","Data":"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"}
Sep 16 19:12:49.824009 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.823986    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:49.846216 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert\") pod \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") "
Sep 16 19:12:49.846355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4k4v\" (UniqueName: \"kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v\") pod \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") "
Sep 16 19:12:49.846355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location\") pod \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") "
Sep 16 19:12:49.846355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config\") pod \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") "
Sep 16 19:12:49.846355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls\") pod \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\" (UID: \"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c\") "
Sep 16 19:12:49.846673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846635    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" (UID: "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:49.846801 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.846781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" (UID: "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:49.847203 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.847182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" (UID: "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.848345 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.848320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" (UID: "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:49.848477 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.848456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v" (OuterVolumeSpecName: "kube-api-access-g4k4v") pod "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" (UID: "d5f74688-aed4-4ea8-a5c0-f66e4c159b1c"). InnerVolumeSpecName "kube-api-access-g4k4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:49.947122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.947038    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.947122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.947066    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.947122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.947080    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.947122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.947091    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k4v\" (UniqueName: \"kubernetes.io/projected/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kube-api-access-g4k4v\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.947122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:49.947104    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:12:50.218682 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.218594    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerID="608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa" exitCode=0
Sep 16 19:12:50.218682 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.218643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerDied","Data":"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"}
Sep 16 19:12:50.218682 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.218657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"
Sep 16 19:12:50.218942 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.218689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg" event={"ID":"d5f74688-aed4-4ea8-a5c0-f66e4c159b1c","Type":"ContainerDied","Data":"451de08ca9119b2dd8a0618043ffbf772c3a8f38163be60d15f25c6b9a428dd1"}
Sep 16 19:12:50.218942 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.218717    2581 scope.go:117] "RemoveContainer" containerID="b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"
Sep 16 19:12:50.234083 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.234062    2581 scope.go:117] "RemoveContainer" containerID="608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"
Sep 16 19:12:50.241693 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.241671    2581 scope.go:117] "RemoveContainer" containerID="958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"
Sep 16 19:12:50.247714 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.247693    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:50.249616 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.249594    2581 scope.go:117] "RemoveContainer" containerID="b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"
Sep 16 19:12:50.249873 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:50.249855    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724\": container with ID starting with b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724 not found: ID does not exist" containerID="b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"
Sep 16 19:12:50.249952 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.249880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724"} err="failed to get container status \"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724\": rpc error: code = NotFound desc = could not find container \"b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724\": container with ID starting with b204e8b51008d2ffbdfd504104190329728bafe86bfd03224d2123af3f6ad724 not found: ID does not exist"
Sep 16 19:12:50.249952 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.249897    2581 scope.go:117] "RemoveContainer" containerID="608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"
Sep 16 19:12:50.250193 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:50.250175    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa\": container with ID starting with 608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa not found: ID does not exist" containerID="608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"
Sep 16 19:12:50.250248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.250198    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa"} err="failed to get container status \"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa\": rpc error: code = NotFound desc = could not find container \"608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa\": container with ID starting with 608d7bbd629f7d7eb2c27b048d222aaf689a95a6ac2b6dc1e2a62e8d4e3b58aa not found: ID does not exist"
Sep 16 19:12:50.250248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.250214    2581 scope.go:117] "RemoveContainer" containerID="958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"
Sep 16 19:12:50.250454 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:50.250438    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4\": container with ID starting with 958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4 not found: ID does not exist" containerID="958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"
Sep 16 19:12:50.250492 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.250461    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4"} err="failed to get container status \"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4\": rpc error: code = NotFound desc = could not find container \"958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4\": container with ID starting with 958d772b28187762769e1678ac8fbab9fe0c4641d6b2c0f714980d9b614d87d4 not found: ID does not exist"
Sep 16 19:12:50.258213 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:50.258189    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg"]
Sep 16 19:12:51.414840 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:51.414806    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" path="/var/lib/kubelet/pods/d5f74688-aed4-4ea8-a5c0-f66e4c159b1c/volumes"
Sep 16 19:12:53.769684 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.769650    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:12:53.770191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kube-rbac-proxy"
Sep 16 19:12:53.770191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770178    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kube-rbac-proxy"
Sep 16 19:12:53.770314 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770201    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container"
Sep 16 19:12:53.770314 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770209    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container"
Sep 16 19:12:53.770314 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770227    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="storage-initializer"
Sep 16 19:12:53.770314 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770236    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="storage-initializer"
Sep 16 19:12:53.770513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770335    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kube-rbac-proxy"
Sep 16 19:12:53.770513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.770346    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f74688-aed4-4ea8-a5c0-f66e4c159b1c" containerName="kserve-container"
Sep 16 19:12:53.775111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.775090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.777359 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.777328    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:12:53.777359 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.777342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:12:53.777519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.777338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:12:53.777637 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.777623    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:53.777684 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.777647    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:12:53.778051 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.778036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:53.778229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.778212    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:12:53.782040 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.782017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:12:53.873512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.873483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpck\" (UniqueName: \"kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.873673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.873525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.873673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.873553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.873673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.873625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.873673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.873662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974350 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpck\" (UniqueName: \"kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:53.974586    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:12:53.974814 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:12:53.974655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls podName:6ecb9a5d-a401-46f3-afa3-7f96fe3e4563 nodeName:}" failed. No retries permitted until 2026-09-16 19:12:54.474636572 +0000 UTC m=+1749.536975814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls") pod "isvc-canary-create-predictor-6cc94c6f4f-lcngg" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:12:53.975064 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.974844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.975205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.975133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.975205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.975133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:53.982383 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:53.982367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpck\" (UniqueName: \"kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:54.478462 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:54.478423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:54.480865 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:54.480838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-lcngg\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:54.685977 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:54.685915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:12:54.962011 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:54.961985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:12:54.964233 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:12:54.964201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecb9a5d_a401_46f3_afa3_7f96fe3e4563.slice/crio-00f8aca0fbc5498b49e0f88922a780f9014cb2e5e2a44a6fcca3a88d17916e2f WatchSource:0}: Error finding container 00f8aca0fbc5498b49e0f88922a780f9014cb2e5e2a44a6fcca3a88d17916e2f: Status 404 returned error can't find the container with id 00f8aca0fbc5498b49e0f88922a780f9014cb2e5e2a44a6fcca3a88d17916e2f
Sep 16 19:12:55.243699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:55.243662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerStarted","Data":"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"}
Sep 16 19:12:55.243861 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:55.243723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerStarted","Data":"00f8aca0fbc5498b49e0f88922a780f9014cb2e5e2a44a6fcca3a88d17916e2f"}
Sep 16 19:12:59.263737 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:59.263652    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerID="b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1" exitCode=0
Sep 16 19:12:59.263737 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:59.263715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerDied","Data":"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"}
Sep 16 19:12:59.264871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:12:59.264855    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:13:13.344478 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:13.344443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerStarted","Data":"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"}
Sep 16 19:13:13.344869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:13.344511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerStarted","Data":"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"}
Sep 16 19:13:13.344869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:13.344813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:13:13.365835 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:13.365785    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podStartSLOduration=3.210230261 podStartE2EDuration="20.36577094s" totalImagesPullingTime="13.336932582s" totalInitContainerRuntime="3.818608097s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:13.36412575 +0000 UTC m=+1768.426465002" watchObservedRunningTime="2026-09-16 19:13:13.36577094 +0000 UTC m=+1768.428110192"
Sep 16 19:13:14.349441 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:14.349409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:13:14.350756 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:14.350725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:13:15.354038 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:15.353996    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:13:20.358742 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:20.358711    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:13:20.359242 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:20.359179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:13:30.359414 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:30.359370    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:13:40.360213 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:40.360120    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:13:50.359284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:13:50.359237    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:14:00.359484 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:00.359443    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:14:10.359457 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:10.359415    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:14:18.123828 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.123790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:14:18.127404 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.127388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.130187 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.130161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 19:14:18.130325 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.130172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 19:14:18.145927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.145899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:14:18.163413 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.163379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.163554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.163447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.163554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.163473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.163554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.163537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q779m\" (UniqueName: \"kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.163688 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.163607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.264968 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.264921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265189 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.264995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265189 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265189 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q779m\" (UniqueName: \"kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265189 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265537 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:18.265514    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:14:18.265577 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265620 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:18.265607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls podName:16e36d45-0a85-478c-8244-e35b54b3cdea nodeName:}" failed. No retries permitted until 2026-09-16 19:14:18.765586587 +0000 UTC m=+1833.827925819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:14:18.265835 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.265873 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.265823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.274283 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.274257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q779m\" (UniqueName: \"kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.770388 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.770352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:18.772753 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:18.772734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:19.037311 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:19.037222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:19.324068 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:19.323924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:14:19.327205 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:14:19.327176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e36d45_0a85_478c_8244_e35b54b3cdea.slice/crio-df44cc6623985c272c1abdb3186b0287d10e1892fcc5eae6fe9c2903fe96aa1a WatchSource:0}: Error finding container df44cc6623985c272c1abdb3186b0287d10e1892fcc5eae6fe9c2903fe96aa1a: Status 404 returned error can't find the container with id df44cc6623985c272c1abdb3186b0287d10e1892fcc5eae6fe9c2903fe96aa1a
Sep 16 19:14:19.641637 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:19.641600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerStarted","Data":"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"}
Sep 16 19:14:19.641837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:19.641662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerStarted","Data":"df44cc6623985c272c1abdb3186b0287d10e1892fcc5eae6fe9c2903fe96aa1a"}
Sep 16 19:14:20.359165 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:20.359121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:14:23.659982 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:23.659950    2581 generic.go:356] "Generic (PLEG): container finished" podID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerID="d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9" exitCode=0
Sep 16 19:14:23.660428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:23.660024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerDied","Data":"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"}
Sep 16 19:14:24.666633 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.666601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerStarted","Data":"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"}
Sep 16 19:14:24.667032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.666661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerStarted","Data":"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"}
Sep 16 19:14:24.667032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.666977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:24.667131 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.667109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:24.668329 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.668306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:24.692452 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:24.692389    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podStartSLOduration=3.054597931 podStartE2EDuration="6.692367859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.637769928s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:24.689917268 +0000 UTC m=+1839.752256520" watchObservedRunningTime="2026-09-16 19:14:24.692367859 +0000 UTC m=+1839.754707112"
Sep 16 19:14:25.672628 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:25.672586    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:26.677211 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:26.677142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:30.359332 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:30.359296    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:14:31.682401 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:31.682371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:14:31.682944 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:31.682915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:33.769401 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:33.769371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:14:33.769778 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:33.769601    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container" containerID="cri-o://68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1" gracePeriod=30
Sep 16 19:14:33.769778 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:33.769628    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kube-rbac-proxy" containerID="cri-o://c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02" gracePeriod=30
Sep 16 19:14:34.714631 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:34.714595    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerID="c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02" exitCode=2
Sep 16 19:14:34.714809 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:34.714651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerDied","Data":"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"}
Sep 16 19:14:35.354717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:35.354674    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 16 19:14:38.719231 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.719205    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:14:38.741213 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.741130    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerID="68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1" exitCode=0
Sep 16 19:14:38.741213 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.741195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerDied","Data":"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"}
Sep 16 19:14:38.741648 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.741208    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"
Sep 16 19:14:38.741648 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.741308    2581 scope.go:117] "RemoveContainer" containerID="c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"
Sep 16 19:14:38.741648 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.741293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg" event={"ID":"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563","Type":"ContainerDied","Data":"00f8aca0fbc5498b49e0f88922a780f9014cb2e5e2a44a6fcca3a88d17916e2f"}
Sep 16 19:14:38.750587 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.750564    2581 scope.go:117] "RemoveContainer" containerID="68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"
Sep 16 19:14:38.758599 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.758573    2581 scope.go:117] "RemoveContainer" containerID="b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"
Sep 16 19:14:38.766260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.766235    2581 scope.go:117] "RemoveContainer" containerID="c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"
Sep 16 19:14:38.766549 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:38.766521    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02\": container with ID starting with c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02 not found: ID does not exist" containerID="c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"
Sep 16 19:14:38.766619 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.766564    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02"} err="failed to get container status \"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02\": rpc error: code = NotFound desc = could not find container \"c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02\": container with ID starting with c8f8166f10330f04f0a228573ca8fc79c3b57033c334c2a6cfebb133133eae02 not found: ID does not exist"
Sep 16 19:14:38.766619 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.766591    2581 scope.go:117] "RemoveContainer" containerID="68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"
Sep 16 19:14:38.766869 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:38.766851    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1\": container with ID starting with 68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1 not found: ID does not exist" containerID="68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"
Sep 16 19:14:38.766917 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.766875    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1"} err="failed to get container status \"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1\": rpc error: code = NotFound desc = could not find container \"68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1\": container with ID starting with 68b93919dba6f0fb422b3d1f810e87705c2976c20b7a926d20625c6ed3c39bf1 not found: ID does not exist"
Sep 16 19:14:38.766917 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.766892    2581 scope.go:117] "RemoveContainer" containerID="b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"
Sep 16 19:14:38.767126 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:38.767110    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1\": container with ID starting with b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1 not found: ID does not exist" containerID="b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"
Sep 16 19:14:38.767216 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.767133    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1"} err="failed to get container status \"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1\": rpc error: code = NotFound desc = could not find container \"b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1\": container with ID starting with b79db45bda5e9c94df1f6cdc75403132fa5c8ef2c7efb10c11755b1c0edcd3f1 not found: ID does not exist"
Sep 16 19:14:38.843535 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert\") pod \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") "
Sep 16 19:14:38.843700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config\") pod \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") "
Sep 16 19:14:38.843700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpck\" (UniqueName: \"kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck\") pod \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") "
Sep 16 19:14:38.843700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location\") pod \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") "
Sep 16 19:14:38.843700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") pod \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\" (UID: \"6ecb9a5d-a401-46f3-afa3-7f96fe3e4563\") "
Sep 16 19:14:38.843964 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:38.844059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:38.844059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.843977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:38.845886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.845858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:38.846034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.845973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck" (OuterVolumeSpecName: "kube-api-access-pgpck") pod "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" (UID: "6ecb9a5d-a401-46f3-afa3-7f96fe3e4563"). InnerVolumeSpecName "kube-api-access-pgpck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:38.928755 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.928724    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:14:38.929205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929187    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container"
Sep 16 19:14:38.929304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929207    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container"
Sep 16 19:14:38.929304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929231    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="storage-initializer"
Sep 16 19:14:38.929304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929239    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="storage-initializer"
Sep 16 19:14:38.929304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929256    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kube-rbac-proxy"
Sep 16 19:14:38.929304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929265    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kube-rbac-proxy"
Sep 16 19:14:38.929566 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929351    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kube-rbac-proxy"
Sep 16 19:14:38.929566 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.929366    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" containerName="kserve-container"
Sep 16 19:14:38.934054 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.934033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:38.936261 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.936234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:14:38.936261 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.936253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:14:38.942057 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.942034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:14:38.944541 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.944520    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.944541 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.944541    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.944657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.944551    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.944657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.944560    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpck\" (UniqueName: \"kubernetes.io/projected/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kube-api-access-pgpck\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.944657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:38.944576    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:14:39.036301 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.036219    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:14:39.039926 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.039909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.042004 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.041977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:14:39.044969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.044944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.045071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.044989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.045177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.045072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.045177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.045120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8nn\" (UniqueName: \"kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.045177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.045166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.049364 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.049341    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:14:39.077391 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.077358    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:14:39.080296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.080270    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg"]
Sep 16 19:14:39.146041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8nn\" (UniqueName: \"kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fglvb\" (UniqueName: \"kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.146257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146685 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146685 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:39.146382    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:14:39.146685 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:39.146462    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls podName:d56118de-96e2-4189-b0de-69d13a5f986b nodeName:}" failed. No retries permitted until 2026-09-16 19:14:39.646441042 +0000 UTC m=+1854.708780276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls") pod "isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:14:39.146685 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.146979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.147015 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.146960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.154329 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.154306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8nn\" (UniqueName: \"kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.246727 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.246688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fglvb\" (UniqueName: \"kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.246727 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.246740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.246967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.246892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.246967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.246941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.247071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.246981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.247133 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:39.247094    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:14:39.247233 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:14:39.247196    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls podName:8ff58bf8-8410-45ca-8bd9-e89118772331 nodeName:}" failed. No retries permitted until 2026-09-16 19:14:39.747141296 +0000 UTC m=+1854.809480531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls") pod "isvc-canary-promote-predictor-b9b98bcd9-tdnsw" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:14:39.247383 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.247359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.247492 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.247476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.247591 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.247572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.255481 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.255459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fglvb\" (UniqueName: \"kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.415271 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.415241    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecb9a5d-a401-46f3-afa3-7f96fe3e4563" path="/var/lib/kubelet/pods/6ecb9a5d-a401-46f3-afa3-7f96fe3e4563/volumes"
Sep 16 19:14:39.650576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.650534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.652919 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.652896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.751270 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.751188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.753608 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.753586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tdnsw\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:39.845585 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.845540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:39.951486 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:39.951456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:40.118446 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.118414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:14:40.121422 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:14:40.121397    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56118de_96e2_4189_b0de_69d13a5f986b.slice/crio-7d69d7e23b32f825cdd4bcfbf750768904ec29ccd50191183486a57dd2687609 WatchSource:0}: Error finding container 7d69d7e23b32f825cdd4bcfbf750768904ec29ccd50191183486a57dd2687609: Status 404 returned error can't find the container with id 7d69d7e23b32f825cdd4bcfbf750768904ec29ccd50191183486a57dd2687609
Sep 16 19:14:40.226320 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.226295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:14:40.229470 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:14:40.229439    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff58bf8_8410_45ca_8bd9_e89118772331.slice/crio-f3529b211f16653972dfa7fa6ed16092f781981b47fb00d2f905ca51410d9dcd WatchSource:0}: Error finding container f3529b211f16653972dfa7fa6ed16092f781981b47fb00d2f905ca51410d9dcd: Status 404 returned error can't find the container with id f3529b211f16653972dfa7fa6ed16092f781981b47fb00d2f905ca51410d9dcd
Sep 16 19:14:40.761801 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.761755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerStarted","Data":"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"}
Sep 16 19:14:40.762483 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.761870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerStarted","Data":"f3529b211f16653972dfa7fa6ed16092f781981b47fb00d2f905ca51410d9dcd"}
Sep 16 19:14:40.763550 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.763520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerStarted","Data":"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"}
Sep 16 19:14:40.763683 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:40.763582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerStarted","Data":"7d69d7e23b32f825cdd4bcfbf750768904ec29ccd50191183486a57dd2687609"}
Sep 16 19:14:41.683205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:41.683146    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:44.783698 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:44.783664    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerID="28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c" exitCode=0
Sep 16 19:14:44.784172 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:44.783742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerDied","Data":"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"}
Sep 16 19:14:44.785436 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:44.785413    2581 generic.go:356] "Generic (PLEG): container finished" podID="d56118de-96e2-4189-b0de-69d13a5f986b" containerID="44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878" exitCode=0
Sep 16 19:14:44.785529 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:44.785494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerDied","Data":"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"}
Sep 16 19:14:45.795228 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.795183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerStarted","Data":"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"}
Sep 16 19:14:45.795701 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.795264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerStarted","Data":"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"}
Sep 16 19:14:45.795701 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.795472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:45.797571 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.797541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerStarted","Data":"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"}
Sep 16 19:14:45.797673 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.797600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerStarted","Data":"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"}
Sep 16 19:14:45.797820 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.797802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:45.797880 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.797854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:45.799666 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.799636    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:14:45.813611 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.813564    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podStartSLOduration=2.780077937 podStartE2EDuration="6.813551547s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.03347361s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:45.812817047 +0000 UTC m=+1860.875156298" watchObservedRunningTime="2026-09-16 19:14:45.813551547 +0000 UTC m=+1860.875890850"
Sep 16 19:14:45.831654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:45.831614    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podStartSLOduration=4.151257454 podStartE2EDuration="7.83160125s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680343796s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:45.829530605 +0000 UTC m=+1860.891869863" watchObservedRunningTime="2026-09-16 19:14:45.83160125 +0000 UTC m=+1860.893940501"
Sep 16 19:14:46.802727 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:46.802692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:46.802727 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:46.802710    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:14:46.803936 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:46.803911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:14:47.806499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:47.806459    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:14:51.683860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:51.683804    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:14:51.807680 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:51.807604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:14:51.808127 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:51.808095    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:14:52.810640 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:52.810607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:14:52.811214 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:14:52.811185    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:01.683699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:01.683640    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:15:01.808270 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:01.808226    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:02.811109 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:02.811062    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:11.683101 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:11.683018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:15:11.809046 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:11.809006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:12.811876 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:12.811833    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:21.682849 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:21.682810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:15:21.808869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:21.808828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:22.811242 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:22.811205    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:31.683216 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:31.683133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:15:31.808974 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:31.808936    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:32.811667 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:32.811628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:41.683363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:41.683319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:15:41.808017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:41.807978    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:42.811898 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:42.811859    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:48.240260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:48.237524    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:15:48.240260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:48.237968    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" containerID="cri-o://7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f" gracePeriod=30
Sep 16 19:15:48.240260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:48.238254    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kube-rbac-proxy" containerID="cri-o://7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431" gracePeriod=30
Sep 16 19:15:49.095663 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:49.095629    2581 generic.go:356] "Generic (PLEG): container finished" podID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerID="7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431" exitCode=2
Sep 16 19:15:49.095812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:49.095697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerDied","Data":"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"}
Sep 16 19:15:51.677859 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:51.677812    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 16 19:15:51.683061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:51.683033    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:15:51.808994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:51.808955    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:15:52.811660 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:52.811621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:15:53.483839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.483811    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:15:53.485911 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.485887    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location\") pod \"16e36d45-0a85-478c-8244-e35b54b3cdea\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") "
Sep 16 19:15:53.486052 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.485939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config\") pod \"16e36d45-0a85-478c-8244-e35b54b3cdea\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") "
Sep 16 19:15:53.486052 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert\") pod \"16e36d45-0a85-478c-8244-e35b54b3cdea\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") "
Sep 16 19:15:53.486052 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") pod \"16e36d45-0a85-478c-8244-e35b54b3cdea\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") "
Sep 16 19:15:53.486256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q779m\" (UniqueName: \"kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m\") pod \"16e36d45-0a85-478c-8244-e35b54b3cdea\" (UID: \"16e36d45-0a85-478c-8244-e35b54b3cdea\") "
Sep 16 19:15:53.486341 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486312    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16e36d45-0a85-478c-8244-e35b54b3cdea" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:53.486440 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486408    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16e36d45-0a85-478c-8244-e35b54b3cdea" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:53.486440 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.486413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16e36d45-0a85-478c-8244-e35b54b3cdea" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:53.488312 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.488278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m" (OuterVolumeSpecName: "kube-api-access-q779m") pod "16e36d45-0a85-478c-8244-e35b54b3cdea" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea"). InnerVolumeSpecName "kube-api-access-q779m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:53.488398 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.488302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16e36d45-0a85-478c-8244-e35b54b3cdea" (UID: "16e36d45-0a85-478c-8244-e35b54b3cdea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:53.587519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.587432    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q779m\" (UniqueName: \"kubernetes.io/projected/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-api-access-q779m\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.587519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.587463    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e36d45-0a85-478c-8244-e35b54b3cdea-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.587519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.587477    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.587519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.587492    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16e36d45-0a85-478c-8244-e35b54b3cdea-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.587519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:53.587507    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e36d45-0a85-478c-8244-e35b54b3cdea-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:15:54.120518 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.120481    2581 generic.go:356] "Generic (PLEG): container finished" podID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerID="7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f" exitCode=0
Sep 16 19:15:54.120991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.120543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerDied","Data":"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"}
Sep 16 19:15:54.120991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.120558    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"
Sep 16 19:15:54.120991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.120614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh" event={"ID":"16e36d45-0a85-478c-8244-e35b54b3cdea","Type":"ContainerDied","Data":"df44cc6623985c272c1abdb3186b0287d10e1892fcc5eae6fe9c2903fe96aa1a"}
Sep 16 19:15:54.120991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.120642    2581 scope.go:117] "RemoveContainer" containerID="7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"
Sep 16 19:15:54.129715 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.129692    2581 scope.go:117] "RemoveContainer" containerID="7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"
Sep 16 19:15:54.136946 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.136929    2581 scope.go:117] "RemoveContainer" containerID="d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"
Sep 16 19:15:54.143052 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.143029    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:15:54.145223 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.145205    2581 scope.go:117] "RemoveContainer" containerID="7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"
Sep 16 19:15:54.145508 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:15:54.145489    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431\": container with ID starting with 7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431 not found: ID does not exist" containerID="7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"
Sep 16 19:15:54.145558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.145517    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431"} err="failed to get container status \"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431\": rpc error: code = NotFound desc = could not find container \"7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431\": container with ID starting with 7bf201eac2b6314db7df403a40cd703817ea5fb2b1e8741f5558239e43766431 not found: ID does not exist"
Sep 16 19:15:54.145558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.145535    2581 scope.go:117] "RemoveContainer" containerID="7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"
Sep 16 19:15:54.145776 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:15:54.145756    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f\": container with ID starting with 7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f not found: ID does not exist" containerID="7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"
Sep 16 19:15:54.145844 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.145786    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f"} err="failed to get container status \"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f\": rpc error: code = NotFound desc = could not find container \"7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f\": container with ID starting with 7ff6f80754d440e6f32df4f9badb39227ff32c0b7e52d29d6fa6d5aed1b14c5f not found: ID does not exist"
Sep 16 19:15:54.145844 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.145807    2581 scope.go:117] "RemoveContainer" containerID="d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"
Sep 16 19:15:54.146108 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:15:54.146087    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9\": container with ID starting with d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9 not found: ID does not exist" containerID="d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"
Sep 16 19:15:54.146171 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.146116    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9"} err="failed to get container status \"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9\": rpc error: code = NotFound desc = could not find container \"d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9\": container with ID starting with d2b2ea2cd4580af2e3e48bd9565c4e0c0de0a0480c9e6da80b41dba82079d2b9 not found: ID does not exist"
Sep 16 19:15:54.146551 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:54.146535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh"]
Sep 16 19:15:55.414597 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:15:55.414563    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" path="/var/lib/kubelet/pods/16e36d45-0a85-478c-8244-e35b54b3cdea/volumes"
Sep 16 19:16:01.809315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:01.809284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:16:02.811986 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:02.811958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:16:09.014171 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:09.014127    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:16:09.014627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:09.014469    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" containerID="cri-o://8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd" gracePeriod=30
Sep 16 19:16:09.014627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:09.014560    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kube-rbac-proxy" containerID="cri-o://8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968" gracePeriod=30
Sep 16 19:16:09.190423 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:09.190390    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerID="8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968" exitCode=2
Sep 16 19:16:09.190574 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:09.190466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerDied","Data":"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"}
Sep 16 19:16:12.807581 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:12.807538    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 19:16:12.811898 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:12.811867    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:16:13.669449 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.669419    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:16:13.764536 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config\") pod \"8ff58bf8-8410-45ca-8bd9-e89118772331\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") "
Sep 16 19:16:13.764536 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fglvb\" (UniqueName: \"kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb\") pod \"8ff58bf8-8410-45ca-8bd9-e89118772331\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") "
Sep 16 19:16:13.764536 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") pod \"8ff58bf8-8410-45ca-8bd9-e89118772331\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") "
Sep 16 19:16:13.764877 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert\") pod \"8ff58bf8-8410-45ca-8bd9-e89118772331\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") "
Sep 16 19:16:13.764877 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location\") pod \"8ff58bf8-8410-45ca-8bd9-e89118772331\" (UID: \"8ff58bf8-8410-45ca-8bd9-e89118772331\") "
Sep 16 19:16:13.764994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ff58bf8-8410-45ca-8bd9-e89118772331" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:13.764994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.764964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ff58bf8-8410-45ca-8bd9-e89118772331" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:13.765085 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.765010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ff58bf8-8410-45ca-8bd9-e89118772331" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:13.766602 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.766580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ff58bf8-8410-45ca-8bd9-e89118772331" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:13.766763 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.766742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb" (OuterVolumeSpecName: "kube-api-access-fglvb") pod "8ff58bf8-8410-45ca-8bd9-e89118772331" (UID: "8ff58bf8-8410-45ca-8bd9-e89118772331"). InnerVolumeSpecName "kube-api-access-fglvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:13.865926 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.865885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.865926 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.865917    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fglvb\" (UniqueName: \"kubernetes.io/projected/8ff58bf8-8410-45ca-8bd9-e89118772331-kube-api-access-fglvb\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.865926 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.865933    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ff58bf8-8410-45ca-8bd9-e89118772331-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.866410 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.865955    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ff58bf8-8410-45ca-8bd9-e89118772331-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.866410 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:13.865968    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ff58bf8-8410-45ca-8bd9-e89118772331-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:16:14.218723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.217735    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerID="8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd" exitCode=0
Sep 16 19:16:14.218723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.217836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerDied","Data":"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"}
Sep 16 19:16:14.218723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.217929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw" event={"ID":"8ff58bf8-8410-45ca-8bd9-e89118772331","Type":"ContainerDied","Data":"f3529b211f16653972dfa7fa6ed16092f781981b47fb00d2f905ca51410d9dcd"}
Sep 16 19:16:14.218723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.217989    2581 scope.go:117] "RemoveContainer" containerID="8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"
Sep 16 19:16:14.218723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.218288    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"
Sep 16 19:16:14.228949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.228929    2581 scope.go:117] "RemoveContainer" containerID="8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"
Sep 16 19:16:14.236699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.236680    2581 scope.go:117] "RemoveContainer" containerID="28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"
Sep 16 19:16:14.244082 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244063    2581 scope.go:117] "RemoveContainer" containerID="8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"
Sep 16 19:16:14.244407 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:16:14.244385    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968\": container with ID starting with 8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968 not found: ID does not exist" containerID="8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"
Sep 16 19:16:14.244509 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244417    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968"} err="failed to get container status \"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968\": rpc error: code = NotFound desc = could not find container \"8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968\": container with ID starting with 8fbd2720d01a7637155e068183bab45468357390a0a68b678f79580e269be968 not found: ID does not exist"
Sep 16 19:16:14.244509 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244441    2581 scope.go:117] "RemoveContainer" containerID="8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"
Sep 16 19:16:14.244594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244513    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:16:14.244721 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:16:14.244703    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd\": container with ID starting with 8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd not found: ID does not exist" containerID="8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"
Sep 16 19:16:14.244765 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244728    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd"} err="failed to get container status \"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd\": rpc error: code = NotFound desc = could not find container \"8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd\": container with ID starting with 8b0db14ddf2cd092879336f89d07cc10502883c8bac000cbaf20925d37c407bd not found: ID does not exist"
Sep 16 19:16:14.244765 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244743    2581 scope.go:117] "RemoveContainer" containerID="28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"
Sep 16 19:16:14.244976 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:16:14.244961    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c\": container with ID starting with 28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c not found: ID does not exist" containerID="28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"
Sep 16 19:16:14.245021 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.244980    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c"} err="failed to get container status \"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c\": rpc error: code = NotFound desc = could not find container \"28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c\": container with ID starting with 28393e6dc08ad777bbdc05e6ef9b5c11f9961cce3a3627aa9ffbde4089f81f7c not found: ID does not exist"
Sep 16 19:16:14.250574 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:14.250554    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw"]
Sep 16 19:16:15.415904 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:15.415873    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" path="/var/lib/kubelet/pods/8ff58bf8-8410-45ca-8bd9-e89118772331/volumes"
Sep 16 19:16:40.027422 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027330    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:16:40.027803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027737    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kube-rbac-proxy"
Sep 16 19:16:40.027803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027753    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kube-rbac-proxy"
Sep 16 19:16:40.027803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027766    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kube-rbac-proxy"
Sep 16 19:16:40.027803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027772    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kube-rbac-proxy"
Sep 16 19:16:40.027803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027796    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027807    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027828    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="storage-initializer"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="storage-initializer"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027847    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027853    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027859    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="storage-initializer"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027867    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="storage-initializer"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027953    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027963    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kserve-container"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027970    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff58bf8-8410-45ca-8bd9-e89118772331" containerName="kube-rbac-proxy"
Sep 16 19:16:40.027994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.027985    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e36d45-0a85-478c-8244-e35b54b3cdea" containerName="kube-rbac-proxy"
Sep 16 19:16:40.031225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.031209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.033642 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.033617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:16:40.033739 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.033655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 19:16:40.042314 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.042291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:16:40.085387 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.085349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.085580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.085407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.085580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.085448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmdwc\" (UniqueName: \"kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.085580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.085493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.085580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.085511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmdwc\" (UniqueName: \"kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186519 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:16:40.186394    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:16:40.186519 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:16:40.186475    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls podName:abfbf70b-035c-4c33-a282-b321e5141201 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:40.686454741 +0000 UTC m=+1975.748793977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" (UID: "abfbf70b-035c-4c33-a282-b321e5141201") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:16:40.186657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186820 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.186860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.186803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.195824 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.195802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmdwc\" (UniqueName: \"kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.690087 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.690050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.692459 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.692435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:40.942081 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:40.941994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:41.215121 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:41.215096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:16:41.216982 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:16:41.216956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfbf70b_035c_4c33_a282_b321e5141201.slice/crio-e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d WatchSource:0}: Error finding container e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d: Status 404 returned error can't find the container with id e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d
Sep 16 19:16:41.339641 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:41.339601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerStarted","Data":"32a2271694dc26d0f002e9aa2d42aa8f0a28431979f2f461ab28b75194c29316"}
Sep 16 19:16:41.339953 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:41.339675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerStarted","Data":"e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d"}
Sep 16 19:16:46.364224 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:46.364188    2581 generic.go:356] "Generic (PLEG): container finished" podID="abfbf70b-035c-4c33-a282-b321e5141201" containerID="32a2271694dc26d0f002e9aa2d42aa8f0a28431979f2f461ab28b75194c29316" exitCode=0
Sep 16 19:16:46.364763 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:46.364268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerDied","Data":"32a2271694dc26d0f002e9aa2d42aa8f0a28431979f2f461ab28b75194c29316"}
Sep 16 19:16:50.389672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.389636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerStarted","Data":"4861391a4aee0026a7283c7e13c763128306ccea40d7903b16b68469cc75254f"}
Sep 16 19:16:50.390183 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.389696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerStarted","Data":"28895bfd6891cec1dc5577210b7987fbc8887a19881831b3241aa59448994953"}
Sep 16 19:16:50.390183 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.389972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:50.390183 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.390139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:50.391256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.391211    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:16:50.413792 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:50.413743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podStartSLOduration=2.249988606 podStartE2EDuration="10.413728729s" totalImagesPullingTime="3.636843547s" totalInitContainerRuntime="4.526896576s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:50.411837144 +0000 UTC m=+1985.474176396" watchObservedRunningTime="2026-09-16 19:16:50.413728729 +0000 UTC m=+1985.476067980"
Sep 16 19:16:51.394612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:51.394568    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:16:56.398962 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:56.398928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:16:56.399462 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:16:56.399434    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:17:06.400078 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:06.400048    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:17:21.038090 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.038059    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:17:21.038511 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.038307    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container" containerID="cri-o://28895bfd6891cec1dc5577210b7987fbc8887a19881831b3241aa59448994953" gracePeriod=30
Sep 16 19:17:21.038511 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.038348    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" containerID="cri-o://4861391a4aee0026a7283c7e13c763128306ccea40d7903b16b68469cc75254f" gracePeriod=30
Sep 16 19:17:21.395320 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.395280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:21.529489 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.529446    2581 generic.go:356] "Generic (PLEG): container finished" podID="abfbf70b-035c-4c33-a282-b321e5141201" containerID="4861391a4aee0026a7283c7e13c763128306ccea40d7903b16b68469cc75254f" exitCode=2
Sep 16 19:17:21.529645 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:21.529522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerDied","Data":"4861391a4aee0026a7283c7e13c763128306ccea40d7903b16b68469cc75254f"}
Sep 16 19:17:26.395853 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:26.395805    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:31.394924 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:31.394882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:31.395324 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:31.395037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:17:36.395218 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:36.395147    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:39.256225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:39.256190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:17:39.256746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:39.256483    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" containerID="cri-o://6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8" gracePeriod=30
Sep 16 19:17:39.256746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:39.256522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kube-rbac-proxy" containerID="cri-o://daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569" gracePeriod=30
Sep 16 19:17:39.613628 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:39.613596    2581 generic.go:356] "Generic (PLEG): container finished" podID="d56118de-96e2-4189-b0de-69d13a5f986b" containerID="daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569" exitCode=2
Sep 16 19:17:39.613808 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:39.613675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerDied","Data":"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"}
Sep 16 19:17:41.394779 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:41.394733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:41.803047 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:41.802962    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:17:41.808030 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:41.808007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:17:44.012930 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.012905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:17:44.116049 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config\") pod \"d56118de-96e2-4189-b0de-69d13a5f986b\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") "
Sep 16 19:17:44.116272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert\") pod \"d56118de-96e2-4189-b0de-69d13a5f986b\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") "
Sep 16 19:17:44.116272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location\") pod \"d56118de-96e2-4189-b0de-69d13a5f986b\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") "
Sep 16 19:17:44.116272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") pod \"d56118de-96e2-4189-b0de-69d13a5f986b\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") "
Sep 16 19:17:44.116272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx8nn\" (UniqueName: \"kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn\") pod \"d56118de-96e2-4189-b0de-69d13a5f986b\" (UID: \"d56118de-96e2-4189-b0de-69d13a5f986b\") "
Sep 16 19:17:44.116463 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d56118de-96e2-4189-b0de-69d13a5f986b" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:44.116510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d56118de-96e2-4189-b0de-69d13a5f986b" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:44.116510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.116462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d56118de-96e2-4189-b0de-69d13a5f986b" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:44.118261 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.118241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn" (OuterVolumeSpecName: "kube-api-access-sx8nn") pod "d56118de-96e2-4189-b0de-69d13a5f986b" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b"). InnerVolumeSpecName "kube-api-access-sx8nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:44.118419 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.118397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d56118de-96e2-4189-b0de-69d13a5f986b" (UID: "d56118de-96e2-4189-b0de-69d13a5f986b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:44.217386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.217336    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8nn\" (UniqueName: \"kubernetes.io/projected/d56118de-96e2-4189-b0de-69d13a5f986b-kube-api-access-sx8nn\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:44.217386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.217378    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:44.217386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.217390    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d56118de-96e2-4189-b0de-69d13a5f986b-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:44.217386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.217400    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d56118de-96e2-4189-b0de-69d13a5f986b-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:44.217649 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.217409    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d56118de-96e2-4189-b0de-69d13a5f986b-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:44.413719 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.413634    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:17:44.413989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.413971    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="storage-initializer"
Sep 16 19:17:44.414035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.413990    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="storage-initializer"
Sep 16 19:17:44.414035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414006    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kube-rbac-proxy"
Sep 16 19:17:44.414035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414012    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kube-rbac-proxy"
Sep 16 19:17:44.414035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414027    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container"
Sep 16 19:17:44.414035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414033    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container"
Sep 16 19:17:44.414218 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414083    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kserve-container"
Sep 16 19:17:44.414218 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.414095    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" containerName="kube-rbac-proxy"
Sep 16 19:17:44.417377 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.417362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.419937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.419915    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:17:44.420022 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.419952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 19:17:44.427943 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.427918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:17:44.457656 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.457626    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:17:44.461219 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.461199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.463537 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.463510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:17:44.474869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.474846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:17:44.519450 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.519450 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.519655 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.519655 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.519655 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbzz\" (UniqueName: \"kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.519655 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.519655 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.519823 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm4v\" (UniqueName: \"kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.519823 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.519823 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.519738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621063 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbzz\" (UniqueName: \"kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm4v\" (UniqueName: \"kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:17:44.621818    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:17:44.621882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls podName:be98f74d-43c4-43ec-ba6f-480bfb249935 nodeName:}" failed. No retries permitted until 2026-09-16 19:17:45.121863687 +0000 UTC m=+2040.184202922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls") pod "isvc-canary-rollback-predictor-665c868b97-8fg87" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:17:44.621983 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.622397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.621986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.622397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.622163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.622397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.622198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.623693 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.623674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.630223 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.630201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbzz\" (UniqueName: \"kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:44.630606 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.630589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm4v\" (UniqueName: \"kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v\") pod \"isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:44.638054 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.638027    2581 generic.go:356] "Generic (PLEG): container finished" podID="d56118de-96e2-4189-b0de-69d13a5f986b" containerID="6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8" exitCode=0
Sep 16 19:17:44.638162 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.638090    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"
Sep 16 19:17:44.638162 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.638103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerDied","Data":"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"}
Sep 16 19:17:44.638250 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.638179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m" event={"ID":"d56118de-96e2-4189-b0de-69d13a5f986b","Type":"ContainerDied","Data":"7d69d7e23b32f825cdd4bcfbf750768904ec29ccd50191183486a57dd2687609"}
Sep 16 19:17:44.638250 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.638209    2581 scope.go:117] "RemoveContainer" containerID="daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"
Sep 16 19:17:44.647561 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.647541    2581 scope.go:117] "RemoveContainer" containerID="6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"
Sep 16 19:17:44.655732 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.655713    2581 scope.go:117] "RemoveContainer" containerID="44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"
Sep 16 19:17:44.661006 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.660979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:17:44.664071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664045    2581 scope.go:117] "RemoveContainer" containerID="daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"
Sep 16 19:17:44.664356 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:17:44.664340    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569\": container with ID starting with daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569 not found: ID does not exist" containerID="daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"
Sep 16 19:17:44.664443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664364    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569"} err="failed to get container status \"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569\": rpc error: code = NotFound desc = could not find container \"daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569\": container with ID starting with daae782d03b095166a37ff07cd92a0b4b5b4874c8b1fbe063054802b5fe9f569 not found: ID does not exist"
Sep 16 19:17:44.664443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664380    2581 scope.go:117] "RemoveContainer" containerID="6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"
Sep 16 19:17:44.664648 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:17:44.664629    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8\": container with ID starting with 6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8 not found: ID does not exist" containerID="6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"
Sep 16 19:17:44.664707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664653    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8"} err="failed to get container status \"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8\": rpc error: code = NotFound desc = could not find container \"6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8\": container with ID starting with 6fb76bb68e8ba7d7e518e9c576feded8bbd0d2f0045a07a068fdbd2a54f8a1f8 not found: ID does not exist"
Sep 16 19:17:44.664707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664668    2581 scope.go:117] "RemoveContainer" containerID="44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"
Sep 16 19:17:44.664890 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:17:44.664875    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878\": container with ID starting with 44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878 not found: ID does not exist" containerID="44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"
Sep 16 19:17:44.664951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.664894    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878"} err="failed to get container status \"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878\": rpc error: code = NotFound desc = could not find container \"44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878\": container with ID starting with 44123ce516f77cda9fe7d2e31eb1a86aa3062fee699c759f054b598ed0584878 not found: ID does not exist"
Sep 16 19:17:44.665205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.665131    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m"]
Sep 16 19:17:44.728280 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:44.728251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:45.004793 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.004768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:17:45.007009 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:17:45.006983    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45cb1bc_fb97_4cd4_be8d_529a875b165a.slice/crio-f48adbd27b39d62e08a7e42aa1759471feebd73b76d35917227a4034ce69f37f WatchSource:0}: Error finding container f48adbd27b39d62e08a7e42aa1759471feebd73b76d35917227a4034ce69f37f: Status 404 returned error can't find the container with id f48adbd27b39d62e08a7e42aa1759471feebd73b76d35917227a4034ce69f37f
Sep 16 19:17:45.126277 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.126232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:45.128741 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.128718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-8fg87\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:45.370876 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.370845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:45.415301 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.415272    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56118de-96e2-4189-b0de-69d13a5f986b" path="/var/lib/kubelet/pods/d56118de-96e2-4189-b0de-69d13a5f986b/volumes"
Sep 16 19:17:45.645811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.645742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:17:45.647417 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.647373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerStarted","Data":"cea17791bc9a04eba2d584cfcbc839226f968fbd66a4307450676a5e50567381"}
Sep 16 19:17:45.647550 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:45.647437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerStarted","Data":"f48adbd27b39d62e08a7e42aa1759471feebd73b76d35917227a4034ce69f37f"}
Sep 16 19:17:45.648166 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:17:45.648129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe98f74d_43c4_43ec_ba6f_480bfb249935.slice/crio-b70b54df7f36b44f8e79fa82a96985b98d4a7017c708f62df09b828805d44810 WatchSource:0}: Error finding container b70b54df7f36b44f8e79fa82a96985b98d4a7017c708f62df09b828805d44810: Status 404 returned error can't find the container with id b70b54df7f36b44f8e79fa82a96985b98d4a7017c708f62df09b828805d44810
Sep 16 19:17:46.395343 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:46.395293    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:46.653735 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:46.653629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerStarted","Data":"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"}
Sep 16 19:17:46.653735 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:46.653711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerStarted","Data":"b70b54df7f36b44f8e79fa82a96985b98d4a7017c708f62df09b828805d44810"}
Sep 16 19:17:48.664399 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:48.664368    2581 generic.go:356] "Generic (PLEG): container finished" podID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerID="cea17791bc9a04eba2d584cfcbc839226f968fbd66a4307450676a5e50567381" exitCode=0
Sep 16 19:17:48.664725 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:48.664451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerDied","Data":"cea17791bc9a04eba2d584cfcbc839226f968fbd66a4307450676a5e50567381"}
Sep 16 19:17:49.671083 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.671053    2581 generic.go:356] "Generic (PLEG): container finished" podID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerID="15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96" exitCode=0
Sep 16 19:17:49.671467 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.671136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerDied","Data":"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"}
Sep 16 19:17:49.673292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.673271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerStarted","Data":"a6a9355aefbedb3c5edbad1867b81f6b2120c0043c0e3f987cff520ba5eb94ae"}
Sep 16 19:17:49.673413 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.673341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerStarted","Data":"29be49aed6203fa7ef2d4dffe3339dfc802d120190ff703439b54a2d02c3f1e1"}
Sep 16 19:17:49.673633 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.673604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:49.673734 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.673657    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:49.675351 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.675327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:17:49.717837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:49.717794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podStartSLOduration=2.145394053 podStartE2EDuration="5.717777358s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.572383305s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:49.71765257 +0000 UTC m=+2044.779991823" watchObservedRunningTime="2026-09-16 19:17:49.717777358 +0000 UTC m=+2044.780116608"
Sep 16 19:17:50.681421 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.681389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerStarted","Data":"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"}
Sep 16 19:17:50.681804 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.681447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerStarted","Data":"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"}
Sep 16 19:17:50.681804 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.681628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:17:50.681918 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.681828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:50.681993 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.681971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:50.683365 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.683338    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:17:50.705083 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:50.705024    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podStartSLOduration=2.801906042 podStartE2EDuration="6.705008773s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.903102731s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:50.703231053 +0000 UTC m=+2045.765570306" watchObservedRunningTime="2026-09-16 19:17:50.705008773 +0000 UTC m=+2045.767348023"
Sep 16 19:17:51.395185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.395126    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:17:51.689730 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.689687    2581 generic.go:356] "Generic (PLEG): container finished" podID="abfbf70b-035c-4c33-a282-b321e5141201" containerID="28895bfd6891cec1dc5577210b7987fbc8887a19881831b3241aa59448994953" exitCode=137
Sep 16 19:17:51.690173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.689724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerDied","Data":"28895bfd6891cec1dc5577210b7987fbc8887a19881831b3241aa59448994953"}
Sep 16 19:17:51.690173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.689798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2" event={"ID":"abfbf70b-035c-4c33-a282-b321e5141201","Type":"ContainerDied","Data":"e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d"}
Sep 16 19:17:51.690173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.689831    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6e471a170a5e8c3cd4df5815815bf1dbe27e1956d6a412a80e9cc67d0f99b5d"
Sep 16 19:17:51.690367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.690325    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:17:51.692756 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.692732    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:17:51.786700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.786659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") pod \"abfbf70b-035c-4c33-a282-b321e5141201\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") "
Sep 16 19:17:51.786700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.786693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmdwc\" (UniqueName: \"kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc\") pod \"abfbf70b-035c-4c33-a282-b321e5141201\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") "
Sep 16 19:17:51.786950 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.786736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location\") pod \"abfbf70b-035c-4c33-a282-b321e5141201\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") "
Sep 16 19:17:51.786950 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.786802    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config\") pod \"abfbf70b-035c-4c33-a282-b321e5141201\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") "
Sep 16 19:17:51.786950 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.786826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert\") pod \"abfbf70b-035c-4c33-a282-b321e5141201\" (UID: \"abfbf70b-035c-4c33-a282-b321e5141201\") "
Sep 16 19:17:51.787328 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.787296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abfbf70b-035c-4c33-a282-b321e5141201" (UID: "abfbf70b-035c-4c33-a282-b321e5141201"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.787679 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.787655    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abfbf70b-035c-4c33-a282-b321e5141201" (UID: "abfbf70b-035c-4c33-a282-b321e5141201"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.789069 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.789036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abfbf70b-035c-4c33-a282-b321e5141201" (UID: "abfbf70b-035c-4c33-a282-b321e5141201"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:51.789377 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.789356    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc" (OuterVolumeSpecName: "kube-api-access-cmdwc") pod "abfbf70b-035c-4c33-a282-b321e5141201" (UID: "abfbf70b-035c-4c33-a282-b321e5141201"). InnerVolumeSpecName "kube-api-access-cmdwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:51.799651 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.799623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abfbf70b-035c-4c33-a282-b321e5141201" (UID: "abfbf70b-035c-4c33-a282-b321e5141201"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.887932 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.887888    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abfbf70b-035c-4c33-a282-b321e5141201-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.887932 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.887930    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmdwc\" (UniqueName: \"kubernetes.io/projected/abfbf70b-035c-4c33-a282-b321e5141201-kube-api-access-cmdwc\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.888121 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.887950    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abfbf70b-035c-4c33-a282-b321e5141201-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.888121 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.887965    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.888121 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:51.887982    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abfbf70b-035c-4c33-a282-b321e5141201-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.696807 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:52.696775    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"
Sep 16 19:17:52.722796 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:52.722764    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:17:52.726929 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:52.726904    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2"]
Sep 16 19:17:53.414927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:53.414893    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfbf70b-035c-4c33-a282-b321e5141201" path="/var/lib/kubelet/pods/abfbf70b-035c-4c33-a282-b321e5141201/volumes"
Sep 16 19:17:55.687794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:55.687764    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:17:55.688365 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:55.688332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:17:56.695099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:56.695071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:17:56.695603 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:17:56.695575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:05.688231 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:05.688185    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:06.695967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:06.695924    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:15.688362 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:15.688277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:16.696442 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:16.696404    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:25.689127 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:25.689081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:26.696145 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:26.696103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:35.688584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:35.688529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:36.696011 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:36.695972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:45.688639 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:45.688588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:46.695582 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:46.695530    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:18:55.688933 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:55.688881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:18:56.695494 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:18:56.695459    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:19:05.688910 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:05.688875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:19:05.822664 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:05.822629    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:19:05.822871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:05.822848    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container" containerID="cri-o://187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15" gracePeriod=30
Sep 16 19:19:05.822972 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:05.822923    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kube-rbac-proxy" containerID="cri-o://aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17" gracePeriod=30
Sep 16 19:19:05.823675 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:05.823278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:19:06.036786 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:06.036750    2581 generic.go:356] "Generic (PLEG): container finished" podID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerID="aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17" exitCode=2
Sep 16 19:19:06.036946 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:06.036819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerDied","Data":"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"}
Sep 16 19:19:06.691062 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:06.691021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:19:10.478319 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.478293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:19:10.650313 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") pod \"be98f74d-43c4-43ec-ba6f-480bfb249935\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") "
Sep 16 19:19:10.650505 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldbzz\" (UniqueName: \"kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz\") pod \"be98f74d-43c4-43ec-ba6f-480bfb249935\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") "
Sep 16 19:19:10.650505 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location\") pod \"be98f74d-43c4-43ec-ba6f-480bfb249935\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") "
Sep 16 19:19:10.650505 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert\") pod \"be98f74d-43c4-43ec-ba6f-480bfb249935\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") "
Sep 16 19:19:10.650505 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config\") pod \"be98f74d-43c4-43ec-ba6f-480bfb249935\" (UID: \"be98f74d-43c4-43ec-ba6f-480bfb249935\") "
Sep 16 19:19:10.650770 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be98f74d-43c4-43ec-ba6f-480bfb249935" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:10.650822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be98f74d-43c4-43ec-ba6f-480bfb249935" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:10.650822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.650806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be98f74d-43c4-43ec-ba6f-480bfb249935" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:10.652538 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.652510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be98f74d-43c4-43ec-ba6f-480bfb249935" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:10.652656 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.652559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz" (OuterVolumeSpecName: "kube-api-access-ldbzz") pod "be98f74d-43c4-43ec-ba6f-480bfb249935" (UID: "be98f74d-43c4-43ec-ba6f-480bfb249935"). InnerVolumeSpecName "kube-api-access-ldbzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:10.751803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.751768    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be98f74d-43c4-43ec-ba6f-480bfb249935-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:10.751803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.751794    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbzz\" (UniqueName: \"kubernetes.io/projected/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-api-access-ldbzz\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:10.751803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.751804    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be98f74d-43c4-43ec-ba6f-480bfb249935-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:10.752045 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.751813    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:10.752045 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:10.751822    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be98f74d-43c4-43ec-ba6f-480bfb249935-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:11.060487 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.060398    2581 generic.go:356] "Generic (PLEG): container finished" podID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerID="187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15" exitCode=0
Sep 16 19:19:11.060487 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.060441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerDied","Data":"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"}
Sep 16 19:19:11.060487 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.060451    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"
Sep 16 19:19:11.060701 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.060501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87" event={"ID":"be98f74d-43c4-43ec-ba6f-480bfb249935","Type":"ContainerDied","Data":"b70b54df7f36b44f8e79fa82a96985b98d4a7017c708f62df09b828805d44810"}
Sep 16 19:19:11.060701 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.060528    2581 scope.go:117] "RemoveContainer" containerID="aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"
Sep 16 19:19:11.069679 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.069662    2581 scope.go:117] "RemoveContainer" containerID="187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"
Sep 16 19:19:11.077049 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.077030    2581 scope.go:117] "RemoveContainer" containerID="15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"
Sep 16 19:19:11.085229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.085205    2581 scope.go:117] "RemoveContainer" containerID="aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"
Sep 16 19:19:11.085473 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:19:11.085454    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17\": container with ID starting with aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17 not found: ID does not exist" containerID="aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"
Sep 16 19:19:11.085556 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.085485    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17"} err="failed to get container status \"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17\": rpc error: code = NotFound desc = could not find container \"aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17\": container with ID starting with aa5a39c4a3ed7fad639f750eded93be8850368717df6e54680dae07ba5442d17 not found: ID does not exist"
Sep 16 19:19:11.085556 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.085509    2581 scope.go:117] "RemoveContainer" containerID="187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"
Sep 16 19:19:11.085771 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:19:11.085753    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15\": container with ID starting with 187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15 not found: ID does not exist" containerID="187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"
Sep 16 19:19:11.085811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.085777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15"} err="failed to get container status \"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15\": rpc error: code = NotFound desc = could not find container \"187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15\": container with ID starting with 187a3b11464c76ed75a0c2a6ed6955cb52552bdb4f16bfb2fcf9cedd09ce0b15 not found: ID does not exist"
Sep 16 19:19:11.085811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.085794    2581 scope.go:117] "RemoveContainer" containerID="15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"
Sep 16 19:19:11.086058 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.086041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:19:11.086111 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:19:11.086060    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96\": container with ID starting with 15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96 not found: ID does not exist" containerID="15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"
Sep 16 19:19:11.086111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.086075    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96"} err="failed to get container status \"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96\": rpc error: code = NotFound desc = could not find container \"15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96\": container with ID starting with 15d5176279025230616a6386ab89c90313e3fb176aaa1d021467e8916fa91e96 not found: ID does not exist"
Sep 16 19:19:11.092452 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.092430    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87"]
Sep 16 19:19:11.414191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:11.414140    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" path="/var/lib/kubelet/pods/be98f74d-43c4-43ec-ba6f-480bfb249935/volumes"
Sep 16 19:19:34.480135 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.480102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:19:34.480759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.480366    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" containerID="cri-o://29be49aed6203fa7ef2d4dffe3339dfc802d120190ff703439b54a2d02c3f1e1" gracePeriod=30
Sep 16 19:19:34.480759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.480466    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kube-rbac-proxy" containerID="cri-o://a6a9355aefbedb3c5edbad1867b81f6b2120c0043c0e3f987cff520ba5eb94ae" gracePeriod=30
Sep 16 19:19:34.572321 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.572275    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:19:34.573117 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573092    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573120    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573166    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="storage-initializer"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573176    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="storage-initializer"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573196    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573205    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container"
Sep 16 19:19:34.573245 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573231    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="storage-initializer"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573259    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="storage-initializer"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kube-rbac-proxy"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kube-rbac-proxy"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573466    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kserve-container"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573485    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be98f74d-43c4-43ec-ba6f-480bfb249935" containerName="kube-rbac-proxy"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573498    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kserve-container"
Sep 16 19:19:34.573576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.573513    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfbf70b-035c-4c33-a282-b321e5141201" containerName="kube-rbac-proxy"
Sep 16 19:19:34.578173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.578134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.581390 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.581359    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:19:34.595122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.595101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:19:34.649026 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.648989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.649026 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.649029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.649317 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.649053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.649317 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.649178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbr8d\" (UniqueName: \"kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.649317 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.649271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750027 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.749927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750027 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.749984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750027 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750352 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750352 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbr8d\" (UniqueName: \"kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750524 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.750777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.750776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.752574 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.752556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.760065 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.760041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbr8d\" (UniqueName: \"kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d\") pod \"isvc-canary-rollback-predictor-665c868b97-tm5sv\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:34.888605 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:34.888567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:35.160086 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.160062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:19:35.162300 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:19:35.162276    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8174cb49_375c_43f1_ac5e_7abe8a6e7c69.slice/crio-dd3d8464390fd01f98a99dcb0f883945e63692581179da7ff7475364e3ba7fc8 WatchSource:0}: Error finding container dd3d8464390fd01f98a99dcb0f883945e63692581179da7ff7475364e3ba7fc8: Status 404 returned error can't find the container with id dd3d8464390fd01f98a99dcb0f883945e63692581179da7ff7475364e3ba7fc8
Sep 16 19:19:35.174247 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.174219    2581 generic.go:356] "Generic (PLEG): container finished" podID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerID="a6a9355aefbedb3c5edbad1867b81f6b2120c0043c0e3f987cff520ba5eb94ae" exitCode=2
Sep 16 19:19:35.174336 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.174284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerDied","Data":"a6a9355aefbedb3c5edbad1867b81f6b2120c0043c0e3f987cff520ba5eb94ae"}
Sep 16 19:19:35.175487 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.175465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerStarted","Data":"dd3d8464390fd01f98a99dcb0f883945e63692581179da7ff7475364e3ba7fc8"}
Sep 16 19:19:35.682573 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.682529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 16 19:19:35.688916 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:35.688880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:19:36.181979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:36.181940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerStarted","Data":"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"}
Sep 16 19:19:39.199370 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.199337    2581 generic.go:356] "Generic (PLEG): container finished" podID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerID="29be49aed6203fa7ef2d4dffe3339dfc802d120190ff703439b54a2d02c3f1e1" exitCode=0
Sep 16 19:19:39.199771 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.199405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerDied","Data":"29be49aed6203fa7ef2d4dffe3339dfc802d120190ff703439b54a2d02c3f1e1"}
Sep 16 19:19:39.201058 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.201033    2581 generic.go:356] "Generic (PLEG): container finished" podID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerID="b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396" exitCode=0
Sep 16 19:19:39.201214 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.201100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerDied","Data":"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"}
Sep 16 19:19:39.330244 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.330214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:19:39.387982 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.387943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls\") pod \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") "
Sep 16 19:19:39.388482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config\") pod \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") "
Sep 16 19:19:39.388482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location\") pod \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") "
Sep 16 19:19:39.388482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert\") pod \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") "
Sep 16 19:19:39.388482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrm4v\" (UniqueName: \"kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v\") pod \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\" (UID: \"c45cb1bc-fb97-4cd4-be8d-529a875b165a\") "
Sep 16 19:19:39.388482 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c45cb1bc-fb97-4cd4-be8d-529a875b165a" (UID: "c45cb1bc-fb97-4cd4-be8d-529a875b165a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:39.388783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c45cb1bc-fb97-4cd4-be8d-529a875b165a" (UID: "c45cb1bc-fb97-4cd4-be8d-529a875b165a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:39.388783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c45cb1bc-fb97-4cd4-be8d-529a875b165a" (UID: "c45cb1bc-fb97-4cd4-be8d-529a875b165a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:39.388783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388652    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:39.388783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388670    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:39.388783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.388685    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:39.390556 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.390531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v" (OuterVolumeSpecName: "kube-api-access-xrm4v") pod "c45cb1bc-fb97-4cd4-be8d-529a875b165a" (UID: "c45cb1bc-fb97-4cd4-be8d-529a875b165a"). InnerVolumeSpecName "kube-api-access-xrm4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:39.390650 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.390529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c45cb1bc-fb97-4cd4-be8d-529a875b165a" (UID: "c45cb1bc-fb97-4cd4-be8d-529a875b165a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:39.489548 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.489512    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrm4v\" (UniqueName: \"kubernetes.io/projected/c45cb1bc-fb97-4cd4-be8d-529a875b165a-kube-api-access-xrm4v\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:39.489548 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:39.489543    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c45cb1bc-fb97-4cd4-be8d-529a875b165a-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:19:40.209300 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.209274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"
Sep 16 19:19:40.209300 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.209282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt" event={"ID":"c45cb1bc-fb97-4cd4-be8d-529a875b165a","Type":"ContainerDied","Data":"f48adbd27b39d62e08a7e42aa1759471feebd73b76d35917227a4034ce69f37f"}
Sep 16 19:19:40.209866 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.209354    2581 scope.go:117] "RemoveContainer" containerID="a6a9355aefbedb3c5edbad1867b81f6b2120c0043c0e3f987cff520ba5eb94ae"
Sep 16 19:19:40.211927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.211901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerStarted","Data":"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"}
Sep 16 19:19:40.212036 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.211958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerStarted","Data":"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"}
Sep 16 19:19:40.212341 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.212325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:40.218023 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.218006    2581 scope.go:117] "RemoveContainer" containerID="29be49aed6203fa7ef2d4dffe3339dfc802d120190ff703439b54a2d02c3f1e1"
Sep 16 19:19:40.225512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.225493    2581 scope.go:117] "RemoveContainer" containerID="cea17791bc9a04eba2d584cfcbc839226f968fbd66a4307450676a5e50567381"
Sep 16 19:19:40.227708 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.227690    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:19:40.231762 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.231740    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt"]
Sep 16 19:19:40.246949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:40.246878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podStartSLOduration=2.480719432 podStartE2EDuration="6.246840033s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766120601s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:40.246511589 +0000 UTC m=+2155.308850840" watchObservedRunningTime="2026-09-16 19:19:40.246840033 +0000 UTC m=+2155.309179284"
Sep 16 19:19:41.219279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:41.219247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:41.220610 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:41.220579    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:19:41.414296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:41.414263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" path="/var/lib/kubelet/pods/c45cb1bc-fb97-4cd4-be8d-529a875b165a/volumes"
Sep 16 19:19:42.224310 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:42.224267    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:19:47.229060 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:47.229026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:19:47.229470 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:47.229351    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:19:53.704841 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.704805    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:19:53.705393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705335    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kube-rbac-proxy"
Sep 16 19:19:53.705393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kube-rbac-proxy"
Sep 16 19:19:53.705393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705378    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="storage-initializer"
Sep 16 19:19:53.705393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705386    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="storage-initializer"
Sep 16 19:19:53.705612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container"
Sep 16 19:19:53.705612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705412    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container"
Sep 16 19:19:53.705612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705507    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kube-rbac-proxy"
Sep 16 19:19:53.705612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.705522    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45cb1bc-fb97-4cd4-be8d-529a875b165a" containerName="kserve-container"
Sep 16 19:19:53.708678 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.708656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.710970 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.710924    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 16 19:19:53.711180 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.711143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 19:19:53.719717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.719692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:19:53.813125 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.813091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.813125 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.813127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.813392 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.813173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75dv\" (UniqueName: \"kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.813392 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.813243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.813392 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.813282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.914691 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.914650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.914882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.914698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.914882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.914734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75dv\" (UniqueName: \"kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.914882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.914775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.914882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.914841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.915203 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.915102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.915672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.915651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.915717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.915651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.917220 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.917202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:53.923304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:53.923279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75dv\" (UniqueName: \"kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv\") pod \"isvc-xgboost-predictor-7566d65bf7-ccl6q\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:54.019456 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:54.019371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:19:54.289556 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:54.289530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:19:54.291400 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:19:54.291370    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb7ddde_ca93_458a_ae98_3edc96a4d44a.slice/crio-9767bf20a778906fa89b7088b1597020aa55104f42d246de110a9d7f83b08202 WatchSource:0}: Error finding container 9767bf20a778906fa89b7088b1597020aa55104f42d246de110a9d7f83b08202: Status 404 returned error can't find the container with id 9767bf20a778906fa89b7088b1597020aa55104f42d246de110a9d7f83b08202
Sep 16 19:19:55.283468 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:55.283430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerStarted","Data":"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"}
Sep 16 19:19:55.283958 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:55.283504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerStarted","Data":"9767bf20a778906fa89b7088b1597020aa55104f42d246de110a9d7f83b08202"}
Sep 16 19:19:57.230212 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:57.230176    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:19:58.304465 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:58.304431    2581 generic.go:356] "Generic (PLEG): container finished" podID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerID="cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506" exitCode=0
Sep 16 19:19:58.304939 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:58.304507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerDied","Data":"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"}
Sep 16 19:19:58.305787 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:19:58.305772    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:20:07.229679 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:07.229636    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:20:16.402787 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:16.402750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerStarted","Data":"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"}
Sep 16 19:20:16.403326 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:16.402823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerStarted","Data":"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"}
Sep 16 19:20:16.403326 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:16.403041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:20:16.421792 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:16.421743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podStartSLOduration=2.384802229 podStartE2EDuration="23.421729102s" totalImagesPullingTime="17.19199782s" totalInitContainerRuntime="3.844929053s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:16.4212624 +0000 UTC m=+2191.483601652" watchObservedRunningTime="2026-09-16 19:20:16.421729102 +0000 UTC m=+2191.484068353"
Sep 16 19:20:17.229886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:17.229846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:20:17.407405 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:17.407370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:20:17.408790 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:17.408753    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:18.411783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:18.411746    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:23.416126 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:23.416096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:20:23.416576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:23.416550    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:27.229704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:27.229665    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:20:33.416956 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:33.416915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:37.229881 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:37.229834    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:20:43.417293 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:43.417254    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:47.230343 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:47.230302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:20:53.417067 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:53.417010    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:20:55.410836 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:20:55.410792    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:21:03.416702 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:03.416661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:21:05.415055 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:05.415028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:21:13.416765 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:13.416672    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:21:14.691751 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:14.691584    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:21:14.692187 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:14.691882    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" containerID="cri-o://40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55" gracePeriod=30
Sep 16 19:21:14.692187 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:14.691955    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kube-rbac-proxy" containerID="cri-o://fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1" gracePeriod=30
Sep 16 19:21:15.411299 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:15.411251    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:21:15.673788 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:15.673706    2581 generic.go:356] "Generic (PLEG): container finished" podID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerID="fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1" exitCode=2
Sep 16 19:21:15.673788 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:15.673746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerDied","Data":"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"}
Sep 16 19:21:17.224716 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:17.224677    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:21:19.339686 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.339661    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:21:19.353203 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353179    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert\") pod \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") "
Sep 16 19:21:19.353316 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config\") pod \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") "
Sep 16 19:21:19.353316 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location\") pod \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") "
Sep 16 19:21:19.353392 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls\") pod \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") "
Sep 16 19:21:19.353392 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbr8d\" (UniqueName: \"kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d\") pod \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\" (UID: \"8174cb49-375c-43f1-ac5e-7abe8a6e7c69\") "
Sep 16 19:21:19.353647 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8174cb49-375c-43f1-ac5e-7abe8a6e7c69" (UID: "8174cb49-375c-43f1-ac5e-7abe8a6e7c69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:19.353722 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8174cb49-375c-43f1-ac5e-7abe8a6e7c69" (UID: "8174cb49-375c-43f1-ac5e-7abe8a6e7c69"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:19.353722 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.353648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8174cb49-375c-43f1-ac5e-7abe8a6e7c69" (UID: "8174cb49-375c-43f1-ac5e-7abe8a6e7c69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:19.355344 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.355319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8174cb49-375c-43f1-ac5e-7abe8a6e7c69" (UID: "8174cb49-375c-43f1-ac5e-7abe8a6e7c69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:19.355473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.355450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d" (OuterVolumeSpecName: "kube-api-access-tbr8d") pod "8174cb49-375c-43f1-ac5e-7abe8a6e7c69" (UID: "8174cb49-375c-43f1-ac5e-7abe8a6e7c69"). InnerVolumeSpecName "kube-api-access-tbr8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:19.454453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.454354    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.454453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.454391    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbr8d\" (UniqueName: \"kubernetes.io/projected/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-api-access-tbr8d\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.454453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.454409    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.454453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.454424    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.454453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.454439    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8174cb49-375c-43f1-ac5e-7abe8a6e7c69-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.694659 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.694624    2581 generic.go:356] "Generic (PLEG): container finished" podID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerID="40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55" exitCode=0
Sep 16 19:21:19.694856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.694695    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"
Sep 16 19:21:19.694856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.694695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerDied","Data":"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"}
Sep 16 19:21:19.694856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.694758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv" event={"ID":"8174cb49-375c-43f1-ac5e-7abe8a6e7c69","Type":"ContainerDied","Data":"dd3d8464390fd01f98a99dcb0f883945e63692581179da7ff7475364e3ba7fc8"}
Sep 16 19:21:19.694856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.694796    2581 scope.go:117] "RemoveContainer" containerID="fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"
Sep 16 19:21:19.703295 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.703278    2581 scope.go:117] "RemoveContainer" containerID="40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"
Sep 16 19:21:19.711144 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.711120    2581 scope.go:117] "RemoveContainer" containerID="b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"
Sep 16 19:21:19.716269 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.716245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:21:19.718669 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.718655    2581 scope.go:117] "RemoveContainer" containerID="fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"
Sep 16 19:21:19.719001 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:19.718980    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1\": container with ID starting with fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1 not found: ID does not exist" containerID="fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"
Sep 16 19:21:19.719093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.719009    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1"} err="failed to get container status \"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1\": rpc error: code = NotFound desc = could not find container \"fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1\": container with ID starting with fc20d64f26ee29f2f74f250ce5966102b5e7b9a27304b249c18623ed45dc94d1 not found: ID does not exist"
Sep 16 19:21:19.719093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.719027    2581 scope.go:117] "RemoveContainer" containerID="40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"
Sep 16 19:21:19.719382 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:19.719364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55\": container with ID starting with 40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55 not found: ID does not exist" containerID="40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"
Sep 16 19:21:19.719437 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.719389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55"} err="failed to get container status \"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55\": rpc error: code = NotFound desc = could not find container \"40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55\": container with ID starting with 40494aa2694a4c057337a866f8857a1af34032e4f88b3c6bd51cdb98473fbd55 not found: ID does not exist"
Sep 16 19:21:19.719437 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.719405    2581 scope.go:117] "RemoveContainer" containerID="b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"
Sep 16 19:21:19.719672 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:19.719652    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396\": container with ID starting with b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396 not found: ID does not exist" containerID="b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"
Sep 16 19:21:19.719713 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.719678    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396"} err="failed to get container status \"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396\": rpc error: code = NotFound desc = could not find container \"b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396\": container with ID starting with b36a41da1d4130d33e1ec20391a2be277ee5ef292c3a6afeb51d5a668b7eb396 not found: ID does not exist"
Sep 16 19:21:19.722626 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.722603    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv"]
Sep 16 19:21:19.892862 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.892824    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:21:19.893226 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="storage-initializer"
Sep 16 19:21:19.893292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893228    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="storage-initializer"
Sep 16 19:21:19.893292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container"
Sep 16 19:21:19.893292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893241    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container"
Sep 16 19:21:19.893292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kube-rbac-proxy"
Sep 16 19:21:19.893292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893257    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kube-rbac-proxy"
Sep 16 19:21:19.893453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893329    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kube-rbac-proxy"
Sep 16 19:21:19.893453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.893340    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" containerName="kserve-container"
Sep 16 19:21:19.897842 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.897825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:19.900362 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.900344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 19:21:19.900837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.900816    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:21:19.908807 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.908784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:21:19.958812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.958785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:19.958982 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.958838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:19.958982 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.958927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8x6\" (UniqueName: \"kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:19.958982 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.958958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:19.959129 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:19.958981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.014925 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.014844    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:21:20.018638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.018622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.021017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.020992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:21:20.029187 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.029146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:21:20.059479 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzx2m\" (UniqueName: \"kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.059629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.059979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8x6\" (UniqueName: \"kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.059979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.059979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.059826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.060114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.060013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.060404 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.060384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.060481 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.060466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.062044 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.062028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.068221 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.068195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8x6\" (UniqueName: \"kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6\") pod \"isvc-canary-stop-v2-predictor-b7c9ff648-65lhc\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.160524 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.160491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.160644 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.160538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.160644 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.160560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzx2m\" (UniqueName: \"kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.160759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.160691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.160841 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.160817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.160910 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:20.160825    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:21:20.160968 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:20.160944    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls podName:8d77a2d9-c5ba-44a3-8641-d304b028f1fe nodeName:}" failed. No retries permitted until 2026-09-16 19:21:20.660922037 +0000 UTC m=+2255.723261268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls") pod "isvc-canary-stop-predictor-7bb9c548b5-48h5k" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:21:20.161195 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.161176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.161353 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.161332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.161393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.161332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.176065 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.176042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzx2m\" (UniqueName: \"kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.207918 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.207887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:20.483200 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.483170    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:21:20.484933 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:21:20.484909    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff416c4_d3e5_44bd_b25b_bf8d6827ed97.slice/crio-290f233ab4d437bedf3f1ae21a93e98ffb4c7bad747548fa1820bfb44a97f0a1 WatchSource:0}: Error finding container 290f233ab4d437bedf3f1ae21a93e98ffb4c7bad747548fa1820bfb44a97f0a1: Status 404 returned error can't find the container with id 290f233ab4d437bedf3f1ae21a93e98ffb4c7bad747548fa1820bfb44a97f0a1
Sep 16 19:21:20.665889 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.665846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.668276 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.668254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-48h5k\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:20.702702 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.702669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerStarted","Data":"3c217f3c60388e510eef1af3dd8e62fa14f0d795d564064dbede1f37c31a2b51"}
Sep 16 19:21:20.702860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.702728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerStarted","Data":"290f233ab4d437bedf3f1ae21a93e98ffb4c7bad747548fa1820bfb44a97f0a1"}
Sep 16 19:21:20.928901 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:20.928857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:21.209751 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:21.209723    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:21:21.212186 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:21:21.212142    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d77a2d9_c5ba_44a3_8641_d304b028f1fe.slice/crio-12caea55885c651bf9fbe07b530abb4083fd3ca8321c652f85baf6abbaab6fa2 WatchSource:0}: Error finding container 12caea55885c651bf9fbe07b530abb4083fd3ca8321c652f85baf6abbaab6fa2: Status 404 returned error can't find the container with id 12caea55885c651bf9fbe07b530abb4083fd3ca8321c652f85baf6abbaab6fa2
Sep 16 19:21:21.414826 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:21.414784    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8174cb49-375c-43f1-ac5e-7abe8a6e7c69" path="/var/lib/kubelet/pods/8174cb49-375c-43f1-ac5e-7abe8a6e7c69/volumes"
Sep 16 19:21:21.709768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:21.709731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerStarted","Data":"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"}
Sep 16 19:21:21.710275 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:21.709802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerStarted","Data":"12caea55885c651bf9fbe07b530abb4083fd3ca8321c652f85baf6abbaab6fa2"}
Sep 16 19:21:23.417295 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:23.417268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:21:23.845781 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:23.845750    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:21:23.846040 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:23.846016    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container" containerID="cri-o://77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b" gracePeriod=30
Sep 16 19:21:23.846138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:23.846053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kube-rbac-proxy" containerID="cri-o://cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1" gracePeriod=30
Sep 16 19:21:24.724769 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:24.724736    2581 generic.go:356] "Generic (PLEG): container finished" podID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerID="cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1" exitCode=2
Sep 16 19:21:24.725173 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:24.724806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerDied","Data":"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"}
Sep 16 19:21:25.731016 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:25.730985    2581 generic.go:356] "Generic (PLEG): container finished" podID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerID="3c217f3c60388e510eef1af3dd8e62fa14f0d795d564064dbede1f37c31a2b51" exitCode=0
Sep 16 19:21:25.731455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:25.731060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerDied","Data":"3c217f3c60388e510eef1af3dd8e62fa14f0d795d564064dbede1f37c31a2b51"}
Sep 16 19:21:25.732825 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:25.732806    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerID="40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304" exitCode=0
Sep 16 19:21:25.732914 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:25.732859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerDied","Data":"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"}
Sep 16 19:21:26.739863 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.739819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerStarted","Data":"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"}
Sep 16 19:21:26.740301 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.739906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerStarted","Data":"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"}
Sep 16 19:21:26.740301 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.740171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:26.742014 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.741989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerStarted","Data":"a8b665f4a0445396d7d4117fc2f73e24049fc02a6a78945b52ff3c8bb007fd4c"}
Sep 16 19:21:26.742126 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.742039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerStarted","Data":"bb8434274970da1df7274a66a3670b0b08bbd9aeae17b58a10f413ed1b2ae65e"}
Sep 16 19:21:26.742229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.742211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:26.742283 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.742257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:26.744012 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.743977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:21:26.760186 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.760128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podStartSLOduration=3.787781101 podStartE2EDuration="7.76011531s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972334209s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:26.759361531 +0000 UTC m=+2261.821700781" watchObservedRunningTime="2026-09-16 19:21:26.76011531 +0000 UTC m=+2261.822454663"
Sep 16 19:21:26.778806 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:26.778760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podStartSLOduration=3.429576197 podStartE2EDuration="7.778744741s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.349168544s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:26.777811071 +0000 UTC m=+2261.840150323" watchObservedRunningTime="2026-09-16 19:21:26.778744741 +0000 UTC m=+2261.841083992"
Sep 16 19:21:27.601432 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.601406    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:21:27.626697 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.626668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location\") pod \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") "
Sep 16 19:21:27.626859 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.626707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls\") pod \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") "
Sep 16 19:21:27.626859 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.626735    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert\") pod \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") "
Sep 16 19:21:27.626859 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.626846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config\") pod \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") "
Sep 16 19:21:27.627029 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.626872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p75dv\" (UniqueName: \"kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv\") pod \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\" (UID: \"6bb7ddde-ca93-458a-ae98-3edc96a4d44a\") "
Sep 16 19:21:27.627099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.627029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bb7ddde-ca93-458a-ae98-3edc96a4d44a" (UID: "6bb7ddde-ca93-458a-ae98-3edc96a4d44a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:27.627201 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.627184    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.627248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.627174    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6bb7ddde-ca93-458a-ae98-3edc96a4d44a" (UID: "6bb7ddde-ca93-458a-ae98-3edc96a4d44a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:27.627248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.627228    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6bb7ddde-ca93-458a-ae98-3edc96a4d44a" (UID: "6bb7ddde-ca93-458a-ae98-3edc96a4d44a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:27.628795 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.628767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bb7ddde-ca93-458a-ae98-3edc96a4d44a" (UID: "6bb7ddde-ca93-458a-ae98-3edc96a4d44a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:27.628896 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.628848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv" (OuterVolumeSpecName: "kube-api-access-p75dv") pod "6bb7ddde-ca93-458a-ae98-3edc96a4d44a" (UID: "6bb7ddde-ca93-458a-ae98-3edc96a4d44a"). InnerVolumeSpecName "kube-api-access-p75dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:27.728395 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.728307    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.728395 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.728339    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.728395 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.728348    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.728395 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.728358    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p75dv\" (UniqueName: \"kubernetes.io/projected/6bb7ddde-ca93-458a-ae98-3edc96a4d44a-kube-api-access-p75dv\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.748203 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748142    2581 generic.go:356] "Generic (PLEG): container finished" podID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerID="77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b" exitCode=0
Sep 16 19:21:27.748562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748218    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"
Sep 16 19:21:27.748562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerDied","Data":"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"}
Sep 16 19:21:27.748562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q" event={"ID":"6bb7ddde-ca93-458a-ae98-3edc96a4d44a","Type":"ContainerDied","Data":"9767bf20a778906fa89b7088b1597020aa55104f42d246de110a9d7f83b08202"}
Sep 16 19:21:27.748562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748337    2581 scope.go:117] "RemoveContainer" containerID="cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"
Sep 16 19:21:27.748994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.748932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:27.749103 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.749058    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:21:27.750001 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.749979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:21:27.757178 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.757143    2581 scope.go:117] "RemoveContainer" containerID="77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"
Sep 16 19:21:27.766512 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.766491    2581 scope.go:117] "RemoveContainer" containerID="cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"
Sep 16 19:21:27.769759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.769736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:21:27.773968 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.773946    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q"]
Sep 16 19:21:27.775930 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.775913    2581 scope.go:117] "RemoveContainer" containerID="cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"
Sep 16 19:21:27.776272 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:27.776250    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1\": container with ID starting with cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1 not found: ID does not exist" containerID="cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"
Sep 16 19:21:27.776360 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.776278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1"} err="failed to get container status \"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1\": rpc error: code = NotFound desc = could not find container \"cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1\": container with ID starting with cb17a1c2c93fe41863689b49937ec00009a1ecff2ef9f2ed742ae4f390190dc1 not found: ID does not exist"
Sep 16 19:21:27.776360 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.776295    2581 scope.go:117] "RemoveContainer" containerID="77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"
Sep 16 19:21:27.776531 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:27.776513    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b\": container with ID starting with 77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b not found: ID does not exist" containerID="77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"
Sep 16 19:21:27.776569 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.776534    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b"} err="failed to get container status \"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b\": rpc error: code = NotFound desc = could not find container \"77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b\": container with ID starting with 77ee9a7112c19f8c8449007aedd424f96fcb2b4d0a40337b6cb790f125f4c73b not found: ID does not exist"
Sep 16 19:21:27.776569 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.776547    2581 scope.go:117] "RemoveContainer" containerID="cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"
Sep 16 19:21:27.776778 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:21:27.776762    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506\": container with ID starting with cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506 not found: ID does not exist" containerID="cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"
Sep 16 19:21:27.776832 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:27.776781    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506"} err="failed to get container status \"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506\": rpc error: code = NotFound desc = could not find container \"cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506\": container with ID starting with cac833ea3f9a19c143205aad312bde4964f64033763bd4a4f2249a3e2803f506 not found: ID does not exist"
Sep 16 19:21:28.755358 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:28.755323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:21:29.415507 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:29.415472    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" path="/var/lib/kubelet/pods/6bb7ddde-ca93-458a-ae98-3edc96a4d44a/volumes"
Sep 16 19:21:32.753298 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:32.753269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:21:32.753914 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:32.753881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:21:33.759459 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:33.759426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:21:33.759935 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:33.759911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:21:42.753979 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:42.753928    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:21:43.759917 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:43.759876    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:21:52.754520 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:52.754483    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:21:53.760845 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:21:53.760809    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:02.753809 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:02.753761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:22:03.759989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:03.759949    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:12.754772 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:12.754728    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:22:13.759967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:13.759924    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:22.754584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:22.754547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:22:23.760814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:23.760776    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:32.754594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:32.754545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:22:33.760858 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:33.760819    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:39.409176 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:39.409073    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:22:43.760299 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:43.760269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:22:49.414055 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:49.414027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:22:49.551664 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:49.551628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:22:49.551890 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:49.551863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" containerID="cri-o://8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d" gracePeriod=30
Sep 16 19:22:49.551952 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:49.551882    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kube-rbac-proxy" containerID="cri-o://683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922" gracePeriod=30
Sep 16 19:22:49.864352 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:49.864319    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:22:50.144457 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:50.144420    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerID="683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922" exitCode=2
Sep 16 19:22:50.144640 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:50.144495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerDied","Data":"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"}
Sep 16 19:22:50.144700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:50.144643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container" containerID="cri-o://bb8434274970da1df7274a66a3670b0b08bbd9aeae17b58a10f413ed1b2ae65e" gracePeriod=30
Sep 16 19:22:50.144700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:50.144648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kube-rbac-proxy" containerID="cri-o://a8b665f4a0445396d7d4117fc2f73e24049fc02a6a78945b52ff3c8bb007fd4c" gracePeriod=30
Sep 16 19:22:51.151517 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:51.151480    2581 generic.go:356] "Generic (PLEG): container finished" podID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerID="a8b665f4a0445396d7d4117fc2f73e24049fc02a6a78945b52ff3c8bb007fd4c" exitCode=2
Sep 16 19:22:51.151909 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:51.151554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerDied","Data":"a8b665f4a0445396d7d4117fc2f73e24049fc02a6a78945b52ff3c8bb007fd4c"}
Sep 16 19:22:52.749369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:52.749327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 16 19:22:53.756016 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:53.755972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 19:22:53.759973 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:53.759948    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:22:54.406502 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.406476    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:22:54.568746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.568656    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config\") pod \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") "
Sep 16 19:22:54.568746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.568698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert\") pod \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") "
Sep 16 19:22:54.568746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.568718    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") pod \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") "
Sep 16 19:22:54.569027 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.568747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location\") pod \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") "
Sep 16 19:22:54.569027 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.568796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzx2m\" (UniqueName: \"kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m\") pod \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\" (UID: \"8d77a2d9-c5ba-44a3-8641-d304b028f1fe\") "
Sep 16 19:22:54.569134 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.569046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d77a2d9-c5ba-44a3-8641-d304b028f1fe" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.569228 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.569130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d77a2d9-c5ba-44a3-8641-d304b028f1fe" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:54.569228 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.569221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d77a2d9-c5ba-44a3-8641-d304b028f1fe" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.570792 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.570769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d77a2d9-c5ba-44a3-8641-d304b028f1fe" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:54.570890 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.570861    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m" (OuterVolumeSpecName: "kube-api-access-zzx2m") pod "8d77a2d9-c5ba-44a3-8641-d304b028f1fe" (UID: "8d77a2d9-c5ba-44a3-8641-d304b028f1fe"). InnerVolumeSpecName "kube-api-access-zzx2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:54.669620 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.669584    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzx2m\" (UniqueName: \"kubernetes.io/projected/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-api-access-zzx2m\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.669620 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.669614    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.669620 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.669624    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.669853 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.669634    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:54.669853 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:54.669643    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d77a2d9-c5ba-44a3-8641-d304b028f1fe-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.172620 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.172590    2581 generic.go:356] "Generic (PLEG): container finished" podID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerID="bb8434274970da1df7274a66a3670b0b08bbd9aeae17b58a10f413ed1b2ae65e" exitCode=0
Sep 16 19:22:55.172968 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.172663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerDied","Data":"bb8434274970da1df7274a66a3670b0b08bbd9aeae17b58a10f413ed1b2ae65e"}
Sep 16 19:22:55.174735 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.174715    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerID="8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d" exitCode=0
Sep 16 19:22:55.174871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.174785    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"
Sep 16 19:22:55.174871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.174793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerDied","Data":"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"}
Sep 16 19:22:55.174871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.174847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k" event={"ID":"8d77a2d9-c5ba-44a3-8641-d304b028f1fe","Type":"ContainerDied","Data":"12caea55885c651bf9fbe07b530abb4083fd3ca8321c652f85baf6abbaab6fa2"}
Sep 16 19:22:55.175276 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.174880    2581 scope.go:117] "RemoveContainer" containerID="683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"
Sep 16 19:22:55.179513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.179494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:22:55.184662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.184476    2581 scope.go:117] "RemoveContainer" containerID="8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"
Sep 16 19:22:55.191948 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.191927    2581 scope.go:117] "RemoveContainer" containerID="40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"
Sep 16 19:22:55.203259 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.203226    2581 scope.go:117] "RemoveContainer" containerID="683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"
Sep 16 19:22:55.203572 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:22:55.203553    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922\": container with ID starting with 683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922 not found: ID does not exist" containerID="683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"
Sep 16 19:22:55.203676 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.203580    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922"} err="failed to get container status \"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922\": rpc error: code = NotFound desc = could not find container \"683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922\": container with ID starting with 683b7c9b9e73ad3e23a28737eb7cbcdfef431c2589146bebbe8301b26037b922 not found: ID does not exist"
Sep 16 19:22:55.203676 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.203596    2581 scope.go:117] "RemoveContainer" containerID="8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"
Sep 16 19:22:55.203951 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:22:55.203895    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d\": container with ID starting with 8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d not found: ID does not exist" containerID="8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"
Sep 16 19:22:55.203951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.203925    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d"} err="failed to get container status \"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d\": rpc error: code = NotFound desc = could not find container \"8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d\": container with ID starting with 8f0e5b0dedaaaaa01820806fe44eb95ed02176c819d72bf9d6ea56830e9f9d1d not found: ID does not exist"
Sep 16 19:22:55.203951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.203944    2581 scope.go:117] "RemoveContainer" containerID="40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"
Sep 16 19:22:55.204473 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:22:55.204351    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304\": container with ID starting with 40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304 not found: ID does not exist" containerID="40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"
Sep 16 19:22:55.204473 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.204392    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304"} err="failed to get container status \"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304\": rpc error: code = NotFound desc = could not find container \"40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304\": container with ID starting with 40738e04d207024ae6c8b66b7bfac7f7e1ab3f77e589f68f8b915806c3aa3304 not found: ID does not exist"
Sep 16 19:22:55.212894 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.212873    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:22:55.217096 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.217074    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k"]
Sep 16 19:22:55.274028 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.273947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb8x6\" (UniqueName: \"kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6\") pod \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") "
Sep 16 19:22:55.274028 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert\") pod \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") "
Sep 16 19:22:55.274279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location\") pod \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") "
Sep 16 19:22:55.274279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274057    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls\") pod \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") "
Sep 16 19:22:55.274279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config\") pod \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\" (UID: \"eff416c4-d3e5-44bd-b25b-bf8d6827ed97\") "
Sep 16 19:22:55.274439 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eff416c4-d3e5-44bd-b25b-bf8d6827ed97" (UID: "eff416c4-d3e5-44bd-b25b-bf8d6827ed97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:55.274506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eff416c4-d3e5-44bd-b25b-bf8d6827ed97" (UID: "eff416c4-d3e5-44bd-b25b-bf8d6827ed97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:55.274506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.274473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eff416c4-d3e5-44bd-b25b-bf8d6827ed97" (UID: "eff416c4-d3e5-44bd-b25b-bf8d6827ed97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:55.276090 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.276069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eff416c4-d3e5-44bd-b25b-bf8d6827ed97" (UID: "eff416c4-d3e5-44bd-b25b-bf8d6827ed97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:55.276225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.276207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6" (OuterVolumeSpecName: "kube-api-access-fb8x6") pod "eff416c4-d3e5-44bd-b25b-bf8d6827ed97" (UID: "eff416c4-d3e5-44bd-b25b-bf8d6827ed97"). InnerVolumeSpecName "kube-api-access-fb8x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:55.374739 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.374701    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb8x6\" (UniqueName: \"kubernetes.io/projected/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-api-access-fb8x6\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.374739 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.374731    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.374739 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.374744    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.374969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.374757    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.374969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.374769    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff416c4-d3e5-44bd-b25b-bf8d6827ed97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.414034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:55.413996    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" path="/var/lib/kubelet/pods/8d77a2d9-c5ba-44a3-8641-d304b028f1fe/volumes"
Sep 16 19:22:56.180782 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.180755    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"
Sep 16 19:22:56.180782 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.180776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc" event={"ID":"eff416c4-d3e5-44bd-b25b-bf8d6827ed97","Type":"ContainerDied","Data":"290f233ab4d437bedf3f1ae21a93e98ffb4c7bad747548fa1820bfb44a97f0a1"}
Sep 16 19:22:56.181345 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.180848    2581 scope.go:117] "RemoveContainer" containerID="a8b665f4a0445396d7d4117fc2f73e24049fc02a6a78945b52ff3c8bb007fd4c"
Sep 16 19:22:56.189236 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.189215    2581 scope.go:117] "RemoveContainer" containerID="bb8434274970da1df7274a66a3670b0b08bbd9aeae17b58a10f413ed1b2ae65e"
Sep 16 19:22:56.197566 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.197532    2581 scope.go:117] "RemoveContainer" containerID="3c217f3c60388e510eef1af3dd8e62fa14f0d795d564064dbede1f37c31a2b51"
Sep 16 19:22:56.200867 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.200845    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:22:56.205043 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:56.205023    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc"]
Sep 16 19:22:57.414977 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:22:57.414945    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" path="/var/lib/kubelet/pods/eff416c4-d3e5-44bd-b25b-bf8d6827ed97/volumes"
Sep 16 19:23:00.143571 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143536    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143857    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="storage-initializer"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143868    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="storage-initializer"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143878    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kube-rbac-proxy"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kube-rbac-proxy"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143898    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143905    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143914    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143924    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kube-rbac-proxy"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143929    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kube-rbac-proxy"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143936    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="storage-initializer"
Sep 16 19:23:00.143940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143942    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="storage-initializer"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143949    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kube-rbac-proxy"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143955    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kube-rbac-proxy"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143965    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="storage-initializer"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143971    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="storage-initializer"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143979    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.143984    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144034    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kserve-container"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144042    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kserve-container"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144049    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kube-rbac-proxy"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144057    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb7ddde-ca93-458a-ae98-3edc96a4d44a" containerName="kube-rbac-proxy"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d77a2d9-c5ba-44a3-8641-d304b028f1fe" containerName="kube-rbac-proxy"
Sep 16 19:23:00.144369 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.144067    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff416c4-d3e5-44bd-b25b-bf8d6827ed97" containerName="kserve-container"
Sep 16 19:23:00.155375 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.155349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.157844 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.157814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:23:00.158005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.157888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:23:00.158078 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.157892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 16 19:23:00.158682 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.158660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:23:00.158864 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.158844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:23:00.158944 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.158850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:23:00.158944 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.158890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 19:23:00.160890 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.160865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:23:00.320163 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.320108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.320355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.320198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.320355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.320287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgt89\" (UniqueName: \"kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.320355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.320319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.320355 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.320340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.421775 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.421688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.421775 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.421723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.421994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.421855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgt89\" (UniqueName: \"kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.421994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.421898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.421994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.421937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.422119 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.422057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.422517 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.422495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.422517 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.422500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.424390 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.424365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.429689 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.429669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgt89\" (UniqueName: \"kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89\") pod \"isvc-lightgbm-predictor-54f85ddf88-fvgb2\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.466720 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.466682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:00.740340 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:00.740311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:23:00.742420 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:23:00.742393    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf65cf63_2533_4458_b99e_dc2314d38c88.slice/crio-508e4147388067dbe57b8ef25b86b4e29a0cd57c8738bbf88954f1f5293208f1 WatchSource:0}: Error finding container 508e4147388067dbe57b8ef25b86b4e29a0cd57c8738bbf88954f1f5293208f1: Status 404 returned error can't find the container with id 508e4147388067dbe57b8ef25b86b4e29a0cd57c8738bbf88954f1f5293208f1
Sep 16 19:23:01.206504 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:01.206467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerStarted","Data":"9bad2e7685998d63d010ba0a8bf6fe081414119fda550ab8899517367042e8d2"}
Sep 16 19:23:01.206889 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:01.206525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerStarted","Data":"508e4147388067dbe57b8ef25b86b4e29a0cd57c8738bbf88954f1f5293208f1"}
Sep 16 19:23:05.228580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:05.228543    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerID="9bad2e7685998d63d010ba0a8bf6fe081414119fda550ab8899517367042e8d2" exitCode=0
Sep 16 19:23:05.228955 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:05.228593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerDied","Data":"9bad2e7685998d63d010ba0a8bf6fe081414119fda550ab8899517367042e8d2"}
Sep 16 19:23:12.267966 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:12.267933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerStarted","Data":"e2ccc6bdb4491340e480671df75e79d8b18f671d3627024724a109171ecfe82f"}
Sep 16 19:23:12.268382 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:12.267995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerStarted","Data":"f9979aa72f5cf645e8151ea1a75e07bd09c567ab4e5f0fbf710f5ce7f823a370"}
Sep 16 19:23:12.268382 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:12.268181    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:12.294185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:12.294120    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podStartSLOduration=2.451598024 podStartE2EDuration="12.294105017s" totalImagesPullingTime="6.084631316s" totalInitContainerRuntime="3.757875677s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:12.291836078 +0000 UTC m=+2367.354175326" watchObservedRunningTime="2026-09-16 19:23:12.294105017 +0000 UTC m=+2367.356444268"
Sep 16 19:23:13.273026 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:13.272986    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:13.274503 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:13.274470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:14.277987 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:14.277943    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:17.618808 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:17.618772    2581 scope.go:117] "RemoveContainer" containerID="32a2271694dc26d0f002e9aa2d42aa8f0a28431979f2f461ab28b75194c29316"
Sep 16 19:23:17.627037 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:17.627020    2581 scope.go:117] "RemoveContainer" containerID="4861391a4aee0026a7283c7e13c763128306ccea40d7903b16b68469cc75254f"
Sep 16 19:23:17.634292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:17.634275    2581 scope.go:117] "RemoveContainer" containerID="28895bfd6891cec1dc5577210b7987fbc8887a19881831b3241aa59448994953"
Sep 16 19:23:19.282516 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.282489    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:23:19.283034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.283008    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:19.486002 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.485959    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:23:19.492112 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.492085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.494678 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.494656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 19:23:19.494807 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.494659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:23:19.500015 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.499985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:23:19.589419 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.589327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5vl\" (UniqueName: \"kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.589419 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.589384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.589623 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.589452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.589623 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.589507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.589623 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.589525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.690805 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.690761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.690805 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.690803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.690853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5vl\" (UniqueName: \"kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.690895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.690938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691229 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:23:19.691196    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 16 19:23:19.691314 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:23:19.691303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls podName:598974df-2403-4a33-8d28-1af9bf673468 nodeName:}" failed. No retries permitted until 2026-09-16 19:23:20.191274727 +0000 UTC m=+2375.253613969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-l49vq" (UID: "598974df-2403-4a33-8d28-1af9bf673468") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 16 19:23:19.691367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.691327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691551 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.691534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.691592 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.691547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:19.700407 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:19.700376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5vl\" (UniqueName: \"kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:20.196819 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:20.196777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:20.199194 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:20.199167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-l49vq\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:20.403790 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:20.403756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:20.677079 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:20.677056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:23:20.679432 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:23:20.679403    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598974df_2403_4a33_8d28_1af9bf673468.slice/crio-83f452ac6a8ce3aa2f1f1c6d56d013f27d7198d2640659eb70d881e08da9452b WatchSource:0}: Error finding container 83f452ac6a8ce3aa2f1f1c6d56d013f27d7198d2640659eb70d881e08da9452b: Status 404 returned error can't find the container with id 83f452ac6a8ce3aa2f1f1c6d56d013f27d7198d2640659eb70d881e08da9452b
Sep 16 19:23:21.310522 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:21.310479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerStarted","Data":"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"}
Sep 16 19:23:21.310730 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:21.310553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerStarted","Data":"83f452ac6a8ce3aa2f1f1c6d56d013f27d7198d2640659eb70d881e08da9452b"}
Sep 16 19:23:25.328491 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:25.328459    2581 generic.go:356] "Generic (PLEG): container finished" podID="598974df-2403-4a33-8d28-1af9bf673468" containerID="e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345" exitCode=0
Sep 16 19:23:25.328967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:25.328530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerDied","Data":"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"}
Sep 16 19:23:26.334988 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:26.334946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerStarted","Data":"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"}
Sep 16 19:23:26.335422 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:26.335026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerStarted","Data":"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"}
Sep 16 19:23:26.335422 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:26.335275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:27.340799 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:27.340771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:27.342357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:27.342327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:23:28.346362 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:28.346323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:23:29.284059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:29.284013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:33.352705 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:33.352672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:23:33.353205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:33.353178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:23:33.374723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:33.374674    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podStartSLOduration=10.434895296 podStartE2EDuration="14.374659908s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939764612s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:26.363757353 +0000 UTC m=+2381.426096603" watchObservedRunningTime="2026-09-16 19:23:33.374659908 +0000 UTC m=+2388.436999159"
Sep 16 19:23:39.283054 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:39.283013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:43.353719 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:43.353682    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:23:49.283907 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:49.283866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:23:53.353122 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:53.353077    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:23:59.283643 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:23:59.283597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:24:03.353682 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:03.353637    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:24:09.283822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:09.283707    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:24:13.353204 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:13.353145    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:24:19.283053 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:19.283012    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:24:23.353420 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:23.353378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:24:29.283797 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:29.283763    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:24:30.226504 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:30.226469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:24:30.226721 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:30.226695    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container" containerID="cri-o://f9979aa72f5cf645e8151ea1a75e07bd09c567ab4e5f0fbf710f5ce7f823a370" gracePeriod=30
Sep 16 19:24:30.226794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:30.226726    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kube-rbac-proxy" containerID="cri-o://e2ccc6bdb4491340e480671df75e79d8b18f671d3627024724a109171ecfe82f" gracePeriod=30
Sep 16 19:24:30.629182 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:30.629130    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerID="e2ccc6bdb4491340e480671df75e79d8b18f671d3627024724a109171ecfe82f" exitCode=2
Sep 16 19:24:30.629638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:30.629180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerDied","Data":"e2ccc6bdb4491340e480671df75e79d8b18f671d3627024724a109171ecfe82f"}
Sep 16 19:24:33.353817 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:33.353789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:24:34.278502 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:34.278457    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:24:35.653746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.653716    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerID="f9979aa72f5cf645e8151ea1a75e07bd09c567ab4e5f0fbf710f5ce7f823a370" exitCode=0
Sep 16 19:24:35.654064 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.653785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerDied","Data":"f9979aa72f5cf645e8151ea1a75e07bd09c567ab4e5f0fbf710f5ce7f823a370"}
Sep 16 19:24:35.671161 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.671126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:24:35.845306 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgt89\" (UniqueName: \"kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89\") pod \"cf65cf63-2533-4458-b99e-dc2314d38c88\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") "
Sep 16 19:24:35.845306 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location\") pod \"cf65cf63-2533-4458-b99e-dc2314d38c88\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") "
Sep 16 19:24:35.845306 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert\") pod \"cf65cf63-2533-4458-b99e-dc2314d38c88\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") "
Sep 16 19:24:35.845588 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config\") pod \"cf65cf63-2533-4458-b99e-dc2314d38c88\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") "
Sep 16 19:24:35.845588 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845353    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls\") pod \"cf65cf63-2533-4458-b99e-dc2314d38c88\" (UID: \"cf65cf63-2533-4458-b99e-dc2314d38c88\") "
Sep 16 19:24:35.845704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf65cf63-2533-4458-b99e-dc2314d38c88" (UID: "cf65cf63-2533-4458-b99e-dc2314d38c88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:35.845748 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf65cf63-2533-4458-b99e-dc2314d38c88" (UID: "cf65cf63-2533-4458-b99e-dc2314d38c88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:35.845783 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.845755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf65cf63-2533-4458-b99e-dc2314d38c88" (UID: "cf65cf63-2533-4458-b99e-dc2314d38c88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:35.847459 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.847438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89" (OuterVolumeSpecName: "kube-api-access-mgt89") pod "cf65cf63-2533-4458-b99e-dc2314d38c88" (UID: "cf65cf63-2533-4458-b99e-dc2314d38c88"). InnerVolumeSpecName "kube-api-access-mgt89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:35.847550 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.847486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf65cf63-2533-4458-b99e-dc2314d38c88" (UID: "cf65cf63-2533-4458-b99e-dc2314d38c88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:35.946296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.946260    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgt89\" (UniqueName: \"kubernetes.io/projected/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-api-access-mgt89\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:35.946296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.946296    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf65cf63-2533-4458-b99e-dc2314d38c88-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:35.946496 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.946311    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:35.946496 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.946326    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf65cf63-2533-4458-b99e-dc2314d38c88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:35.946496 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:35.946341    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf65cf63-2533-4458-b99e-dc2314d38c88-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:36.661079 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.661041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2" event={"ID":"cf65cf63-2533-4458-b99e-dc2314d38c88","Type":"ContainerDied","Data":"508e4147388067dbe57b8ef25b86b4e29a0cd57c8738bbf88954f1f5293208f1"}
Sep 16 19:24:36.661548 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.661121    2581 scope.go:117] "RemoveContainer" containerID="e2ccc6bdb4491340e480671df75e79d8b18f671d3627024724a109171ecfe82f"
Sep 16 19:24:36.661548 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.661058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"
Sep 16 19:24:36.669407 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.669391    2581 scope.go:117] "RemoveContainer" containerID="f9979aa72f5cf645e8151ea1a75e07bd09c567ab4e5f0fbf710f5ce7f823a370"
Sep 16 19:24:36.676967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.676945    2581 scope.go:117] "RemoveContainer" containerID="9bad2e7685998d63d010ba0a8bf6fe081414119fda550ab8899517367042e8d2"
Sep 16 19:24:36.682871 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.682844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:24:36.689061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:36.689039    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2"]
Sep 16 19:24:37.414927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:37.414894    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" path="/var/lib/kubelet/pods/cf65cf63-2533-4458-b99e-dc2314d38c88/volumes"
Sep 16 19:24:39.557953 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:39.557911    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:24:39.558424 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:39.558243    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container" containerID="cri-o://1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f" gracePeriod=30
Sep 16 19:24:39.558424 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:39.558280    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kube-rbac-proxy" containerID="cri-o://5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6" gracePeriod=30
Sep 16 19:24:39.677059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:39.677025    2581 generic.go:356] "Generic (PLEG): container finished" podID="598974df-2403-4a33-8d28-1af9bf673468" containerID="5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6" exitCode=2
Sep 16 19:24:39.677232 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:39.677097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerDied","Data":"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"}
Sep 16 19:24:43.303705 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.303681    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:24:43.412748 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.412714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location\") pod \"598974df-2403-4a33-8d28-1af9bf673468\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") "
Sep 16 19:24:43.412959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.412766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config\") pod \"598974df-2403-4a33-8d28-1af9bf673468\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") "
Sep 16 19:24:43.412959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.412810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert\") pod \"598974df-2403-4a33-8d28-1af9bf673468\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") "
Sep 16 19:24:43.412959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.412837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm5vl\" (UniqueName: \"kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl\") pod \"598974df-2403-4a33-8d28-1af9bf673468\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") "
Sep 16 19:24:43.412959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.412899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") pod \"598974df-2403-4a33-8d28-1af9bf673468\" (UID: \"598974df-2403-4a33-8d28-1af9bf673468\") "
Sep 16 19:24:43.413183 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.413112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "598974df-2403-4a33-8d28-1af9bf673468" (UID: "598974df-2403-4a33-8d28-1af9bf673468"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:43.413255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.413229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "598974df-2403-4a33-8d28-1af9bf673468" (UID: "598974df-2403-4a33-8d28-1af9bf673468"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:43.413307 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.413279    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "598974df-2403-4a33-8d28-1af9bf673468" (UID: "598974df-2403-4a33-8d28-1af9bf673468"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:43.415035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.415012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl" (OuterVolumeSpecName: "kube-api-access-tm5vl") pod "598974df-2403-4a33-8d28-1af9bf673468" (UID: "598974df-2403-4a33-8d28-1af9bf673468"). InnerVolumeSpecName "kube-api-access-tm5vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:43.415169 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.415046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "598974df-2403-4a33-8d28-1af9bf673468" (UID: "598974df-2403-4a33-8d28-1af9bf673468"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:43.513828 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.513792    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.513828 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.513821    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5vl\" (UniqueName: \"kubernetes.io/projected/598974df-2403-4a33-8d28-1af9bf673468-kube-api-access-tm5vl\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.513828 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.513832    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598974df-2403-4a33-8d28-1af9bf673468-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.513828 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.513841    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598974df-2403-4a33-8d28-1af9bf673468-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.514243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.513851    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598974df-2403-4a33-8d28-1af9bf673468-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.699924 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.699838    2581 generic.go:356] "Generic (PLEG): container finished" podID="598974df-2403-4a33-8d28-1af9bf673468" containerID="1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f" exitCode=0
Sep 16 19:24:43.699924 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.699890    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"
Sep 16 19:24:43.699924 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.699903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerDied","Data":"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"}
Sep 16 19:24:43.700138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.699958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq" event={"ID":"598974df-2403-4a33-8d28-1af9bf673468","Type":"ContainerDied","Data":"83f452ac6a8ce3aa2f1f1c6d56d013f27d7198d2640659eb70d881e08da9452b"}
Sep 16 19:24:43.700138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.699987    2581 scope.go:117] "RemoveContainer" containerID="5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"
Sep 16 19:24:43.708350 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.708325    2581 scope.go:117] "RemoveContainer" containerID="1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"
Sep 16 19:24:43.715920 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.715903    2581 scope.go:117] "RemoveContainer" containerID="e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"
Sep 16 19:24:43.723093 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.723070    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:24:43.723803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.723787    2581 scope.go:117] "RemoveContainer" containerID="5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"
Sep 16 19:24:43.724098 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:24:43.724071    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6\": container with ID starting with 5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6 not found: ID does not exist" containerID="5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"
Sep 16 19:24:43.724195 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.724100    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6"} err="failed to get container status \"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6\": rpc error: code = NotFound desc = could not find container \"5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6\": container with ID starting with 5eaf9e8ff96fb2be4e938e8af9ae173c45e5bd640c45b772f336772940e2cad6 not found: ID does not exist"
Sep 16 19:24:43.724195 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.724117    2581 scope.go:117] "RemoveContainer" containerID="1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"
Sep 16 19:24:43.724397 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:24:43.724381    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f\": container with ID starting with 1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f not found: ID does not exist" containerID="1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"
Sep 16 19:24:43.724443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.724401    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f"} err="failed to get container status \"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f\": rpc error: code = NotFound desc = could not find container \"1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f\": container with ID starting with 1a017d0825211e30e907ad4584844fe599a1c6fb7ee410cb7d534fd3150c809f not found: ID does not exist"
Sep 16 19:24:43.724443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.724413    2581 scope.go:117] "RemoveContainer" containerID="e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"
Sep 16 19:24:43.724637 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:24:43.724615    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345\": container with ID starting with e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345 not found: ID does not exist" containerID="e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"
Sep 16 19:24:43.724684 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.724652    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345"} err="failed to get container status \"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345\": rpc error: code = NotFound desc = could not find container \"e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345\": container with ID starting with e16ec8da49e2a17d3074c31196d2092345133fa5b271634d77e8e9c590908345 not found: ID does not exist"
Sep 16 19:24:43.729471 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:43.729451    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq"]
Sep 16 19:24:45.414290 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:24:45.414261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598974df-2403-4a33-8d28-1af9bf673468" path="/var/lib/kubelet/pods/598974df-2403-4a33-8d28-1af9bf673468/volumes"
Sep 16 19:25:50.010897 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.010821    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011163    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011175    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="storage-initializer"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="storage-initializer"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011200    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011206    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011214    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011220    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011225    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="storage-initializer"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011230    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="storage-initializer"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011241    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011246    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011306    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kube-rbac-proxy"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011317    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf65cf63-2533-4458-b99e-dc2314d38c88" containerName="kserve-container"
Sep 16 19:25:50.011363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.011323    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="598974df-2403-4a33-8d28-1af9bf673468" containerName="kserve-container"
Sep 16 19:25:50.014479 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.014461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.017084 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017055    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 19:25:50.017293 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:25:50.017379 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017348    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:50.017379 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:25:50.017499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:50.017499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:25:50.017614 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.017495    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:50.024822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.024801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:25:50.056838 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.056811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgkx\" (UniqueName: \"kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.056981 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.056843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.056981 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.056878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.056981 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.056923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.057089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.056982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.157767 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.157731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgkx\" (UniqueName: \"kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.157882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.157771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.157882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.157831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.157882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.157850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.157882 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.157878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.158097 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:25:50.158074    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:25:50.158199 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:25:50.158169    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls podName:98566925-8bbc-4a3d-9293-a5a4bca7afa2 nodeName:}" failed. No retries permitted until 2026-09-16 19:25:50.658134987 +0000 UTC m=+2525.720474240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-8t6kd" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:25:50.158328 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.158310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.158703 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.158684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.158741 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.158684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.167022 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.166988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgkx\" (UniqueName: \"kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.662178 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:50.662131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:50.662359 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:25:50.662281    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:25:50.662359 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:25:50.662334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls podName:98566925-8bbc-4a3d-9293-a5a4bca7afa2 nodeName:}" failed. No retries permitted until 2026-09-16 19:25:51.662319495 +0000 UTC m=+2526.724658724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-8t6kd" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:25:51.669234 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:51.669181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:51.671575 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:51.671550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-8t6kd\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:51.825396 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:51.825354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:52.097788 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:52.097761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:25:52.100393 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:25:52.100368    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98566925_8bbc_4a3d_9293_a5a4bca7afa2.slice/crio-0da259c1b51845dc86684e828f93bbe7b8ef5c5bcdfe1c5f12864f3df9c53a27 WatchSource:0}: Error finding container 0da259c1b51845dc86684e828f93bbe7b8ef5c5bcdfe1c5f12864f3df9c53a27: Status 404 returned error can't find the container with id 0da259c1b51845dc86684e828f93bbe7b8ef5c5bcdfe1c5f12864f3df9c53a27
Sep 16 19:25:53.004509 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:53.004471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerStarted","Data":"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"}
Sep 16 19:25:53.004886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:53.004537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerStarted","Data":"0da259c1b51845dc86684e828f93bbe7b8ef5c5bcdfe1c5f12864f3df9c53a27"}
Sep 16 19:25:56.018667 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:56.018632    2581 generic.go:356] "Generic (PLEG): container finished" podID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerID="a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e" exitCode=0
Sep 16 19:25:56.019030 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:56.018708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerDied","Data":"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"}
Sep 16 19:25:57.024902 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:57.024863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerStarted","Data":"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"}
Sep 16 19:25:57.025297 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:57.024921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerStarted","Data":"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"}
Sep 16 19:25:57.025297 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:57.025094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:57.051927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:57.051876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podStartSLOduration=4.374794168 podStartE2EDuration="8.051861551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677067383s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:57.05046027 +0000 UTC m=+2532.112799520" watchObservedRunningTime="2026-09-16 19:25:57.051861551 +0000 UTC m=+2532.114200801"
Sep 16 19:25:58.029984 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:58.029948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:25:58.031389 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:58.031359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:25:59.034625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:25:59.034584    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:04.039333 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:04.039299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:26:04.039794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:04.039766    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:14.039757 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:14.039718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:24.040435 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:24.040395    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:34.040390 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:34.040352    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:44.040208 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:44.040146    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:26:54.039920 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:26:54.039877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:27:04.040327 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:04.040292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:27:06.518869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.518830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:06.522420 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.522395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.524674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.524651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:27:06.524992 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.524976    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 19:27:06.533922 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.533899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:06.606604 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.606573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.606794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.606630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.606794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.606709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.606794 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.606776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.606922 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.606810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8d7\" (UniqueName: \"kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707429 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707624 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707624 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707730 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707730 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8d7\" (UniqueName: \"kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.707814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.707796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.708238 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.708213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.708327 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.708240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.710109 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.710081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.716368 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.716345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8d7\" (UniqueName: \"kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:06.833800 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:06.833716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:07.112430 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:07.112395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:07.115422 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:27:07.115393    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda937e7d8_6979_45f9_bf70_1aa7b5a6678d.slice/crio-18d99a0409dcdd675b220a0c3ec63cfc306190b3bbb80e338882e698bb864440 WatchSource:0}: Error finding container 18d99a0409dcdd675b220a0c3ec63cfc306190b3bbb80e338882e698bb864440: Status 404 returned error can't find the container with id 18d99a0409dcdd675b220a0c3ec63cfc306190b3bbb80e338882e698bb864440
Sep 16 19:27:07.340831 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:07.340793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerStarted","Data":"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"}
Sep 16 19:27:07.341025 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:07.340849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerStarted","Data":"18d99a0409dcdd675b220a0c3ec63cfc306190b3bbb80e338882e698bb864440"}
Sep 16 19:27:10.106598 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:10.106527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:27:10.107019 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:10.106776    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container" containerID="cri-o://51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75" gracePeriod=30
Sep 16 19:27:10.107019 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:10.106859    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kube-rbac-proxy" containerID="cri-o://4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91" gracePeriod=30
Sep 16 19:27:10.361707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:10.361618    2581 generic.go:356] "Generic (PLEG): container finished" podID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerID="4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91" exitCode=2
Sep 16 19:27:10.361707 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:10.361686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerDied","Data":"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"}
Sep 16 19:27:11.368178 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:11.368071    2581 generic.go:356] "Generic (PLEG): container finished" podID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerID="ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466" exitCode=0
Sep 16 19:27:11.368178 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:11.368143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerDied","Data":"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"}
Sep 16 19:27:12.375706 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:12.375665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerStarted","Data":"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"}
Sep 16 19:27:12.376200 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:12.375736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerStarted","Data":"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"}
Sep 16 19:27:12.376200 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:12.375957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:12.396704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:12.396651    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" podStartSLOduration=2.4629229280000002 podStartE2EDuration="6.396635295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.933712367s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:12.394585497 +0000 UTC m=+2607.456924749" watchObservedRunningTime="2026-09-16 19:27:12.396635295 +0000 UTC m=+2607.458974546"
Sep 16 19:27:13.380837 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.380805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:13.382296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.382270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:27:13.772711 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.772687    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:27:13.864894 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.864858    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") pod \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") "
Sep 16 19:27:13.865098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.864921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config\") pod \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") "
Sep 16 19:27:13.865098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.864965    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location\") pod \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") "
Sep 16 19:27:13.865098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.864996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hgkx\" (UniqueName: \"kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx\") pod \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") "
Sep 16 19:27:13.865098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.865025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert\") pod \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\" (UID: \"98566925-8bbc-4a3d-9293-a5a4bca7afa2\") "
Sep 16 19:27:13.865363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.865337    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98566925-8bbc-4a3d-9293-a5a4bca7afa2" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:13.865363 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.865352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98566925-8bbc-4a3d-9293-a5a4bca7afa2" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:13.865480 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.865434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98566925-8bbc-4a3d-9293-a5a4bca7afa2" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:13.867107 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.867061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx" (OuterVolumeSpecName: "kube-api-access-2hgkx") pod "98566925-8bbc-4a3d-9293-a5a4bca7afa2" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2"). InnerVolumeSpecName "kube-api-access-2hgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:13.867230 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.867133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98566925-8bbc-4a3d-9293-a5a4bca7afa2" (UID: "98566925-8bbc-4a3d-9293-a5a4bca7afa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:13.966197 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.966107    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98566925-8bbc-4a3d-9293-a5a4bca7afa2-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:13.966197 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.966134    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:13.966197 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.966145    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:13.966197 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.966170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hgkx\" (UniqueName: \"kubernetes.io/projected/98566925-8bbc-4a3d-9293-a5a4bca7afa2-kube-api-access-2hgkx\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:13.966197 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:13.966180    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98566925-8bbc-4a3d-9293-a5a4bca7afa2-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.388524 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.388483    2581 generic.go:356] "Generic (PLEG): container finished" podID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerID="51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75" exitCode=0
Sep 16 19:27:14.388986 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.388534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"
Sep 16 19:27:14.388986 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.388556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerDied","Data":"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"}
Sep 16 19:27:14.388986 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.388628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd" event={"ID":"98566925-8bbc-4a3d-9293-a5a4bca7afa2","Type":"ContainerDied","Data":"0da259c1b51845dc86684e828f93bbe7b8ef5c5bcdfe1c5f12864f3df9c53a27"}
Sep 16 19:27:14.388986 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.388647    2581 scope.go:117] "RemoveContainer" containerID="4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"
Sep 16 19:27:14.389235 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.389210    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:27:14.398086 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.398070    2581 scope.go:117] "RemoveContainer" containerID="51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"
Sep 16 19:27:14.406674 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.406651    2581 scope.go:117] "RemoveContainer" containerID="a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"
Sep 16 19:27:14.411946 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.411924    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:27:14.415526 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.415503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd"]
Sep 16 19:27:14.415590 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.415529    2581 scope.go:117] "RemoveContainer" containerID="4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"
Sep 16 19:27:14.415828 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:14.415810    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91\": container with ID starting with 4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91 not found: ID does not exist" containerID="4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"
Sep 16 19:27:14.415891 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.415841    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91"} err="failed to get container status \"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91\": rpc error: code = NotFound desc = could not find container \"4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91\": container with ID starting with 4ba9d183984aaa618b1d4e01cdd38bfd43119d3f691f6af23cc1dcac64aafb91 not found: ID does not exist"
Sep 16 19:27:14.415891 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.415866    2581 scope.go:117] "RemoveContainer" containerID="51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"
Sep 16 19:27:14.416170 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:14.416133    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75\": container with ID starting with 51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75 not found: ID does not exist" containerID="51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"
Sep 16 19:27:14.416232 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.416170    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75"} err="failed to get container status \"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75\": rpc error: code = NotFound desc = could not find container \"51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75\": container with ID starting with 51f32ba80a716564c0370f0281fac6e322cbf9c262268bf63cacd1feb0c2fa75 not found: ID does not exist"
Sep 16 19:27:14.416232 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.416186    2581 scope.go:117] "RemoveContainer" containerID="a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"
Sep 16 19:27:14.416463 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:14.416446    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e\": container with ID starting with a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e not found: ID does not exist" containerID="a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"
Sep 16 19:27:14.416515 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:14.416469    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e"} err="failed to get container status \"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e\": rpc error: code = NotFound desc = could not find container \"a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e\": container with ID starting with a7dbb91603c270121adc406100a1beaa48202a0624266811edc29942961fd15e not found: ID does not exist"
Sep 16 19:27:15.317412 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.317370    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:27:15.317980 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.317958    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="storage-initializer"
Sep 16 19:27:15.318041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.317984    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="storage-initializer"
Sep 16 19:27:15.318041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318015    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kube-rbac-proxy"
Sep 16 19:27:15.318041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318024    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kube-rbac-proxy"
Sep 16 19:27:15.318041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container"
Sep 16 19:27:15.318177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318047    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container"
Sep 16 19:27:15.318177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318137    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kserve-container"
Sep 16 19:27:15.318177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.318166    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" containerName="kube-rbac-proxy"
Sep 16 19:27:15.323307 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.323286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.325596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.325573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 16 19:27:15.325714 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.325598    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 16 19:27:15.325714 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.325654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:27:15.334591 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.334568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:27:15.414279 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.414245    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98566925-8bbc-4a3d-9293-a5a4bca7afa2" path="/var/lib/kubelet/pods/98566925-8bbc-4a3d-9293-a5a4bca7afa2/volumes"
Sep 16 19:27:15.482796 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.482759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gxb\" (UniqueName: \"kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.482796 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.482799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.483071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.482961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.483071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.483030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.583928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.583838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.583928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.583893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.584141 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.583946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gxb\" (UniqueName: \"kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.584141 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.583970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.584508 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.584470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.584630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.584611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.586419 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.586399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.593864 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.593840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gxb\" (UniqueName: \"kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb\") pod \"isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.634718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.634673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:15.910650 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:15.910622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:27:15.913271 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:27:15.913227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2edc88_68ce_4ea9_a6c1_af195a87f525.slice/crio-069ab23bf53b8b591b0d6babcacaa65326f52498306c3dabb8be43b8586ecf20 WatchSource:0}: Error finding container 069ab23bf53b8b591b0d6babcacaa65326f52498306c3dabb8be43b8586ecf20: Status 404 returned error can't find the container with id 069ab23bf53b8b591b0d6babcacaa65326f52498306c3dabb8be43b8586ecf20
Sep 16 19:27:16.401697 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:16.401664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerStarted","Data":"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"}
Sep 16 19:27:16.401918 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:16.401737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerStarted","Data":"069ab23bf53b8b591b0d6babcacaa65326f52498306c3dabb8be43b8586ecf20"}
Sep 16 19:27:17.408125 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:17.408091    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerID="7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637" exitCode=0
Sep 16 19:27:17.408566 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:17.408185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerDied","Data":"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"}
Sep 16 19:27:18.416681 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:18.416643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerStarted","Data":"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"}
Sep 16 19:27:18.417174 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:18.416710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerStarted","Data":"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"}
Sep 16 19:27:18.417174 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:18.416842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:18.442698 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:18.442645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podStartSLOduration=2.4292414 podStartE2EDuration="3.442628036s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.013386636s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:18.441771149 +0000 UTC m=+2613.504110401" watchObservedRunningTime="2026-09-16 19:27:18.442628036 +0000 UTC m=+2613.504967287"
Sep 16 19:27:19.394284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:19.394255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:19.395305 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:19.395278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:19.420940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:19.420910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:19.422324 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:19.422300    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:20.426407 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:20.426369    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:25.431595 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:25.431564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:27:25.432089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:25.432062    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:26.630552 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:26.630521    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:26.630904 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:26.630740    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container" containerID="cri-o://f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79" gracePeriod=30
Sep 16 19:27:26.630904 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:26.630807    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kube-rbac-proxy" containerID="cri-o://9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40" gracePeriod=30
Sep 16 19:27:27.274397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.274374    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:27.383764 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.383663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls\") pod \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") "
Sep 16 19:27:27.383929 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.383814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location\") pod \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") "
Sep 16 19:27:27.383929 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.383876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert\") pod \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") "
Sep 16 19:27:27.384022 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.383920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config\") pod \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") "
Sep 16 19:27:27.384022 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.383986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s8d7\" (UniqueName: \"kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7\") pod \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\" (UID: \"a937e7d8-6979-45f9-bf70-1aa7b5a6678d\") "
Sep 16 19:27:27.384304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.384199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a937e7d8-6979-45f9-bf70-1aa7b5a6678d" (UID: "a937e7d8-6979-45f9-bf70-1aa7b5a6678d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:27.384391 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.384293    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a937e7d8-6979-45f9-bf70-1aa7b5a6678d" (UID: "a937e7d8-6979-45f9-bf70-1aa7b5a6678d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:27.384391 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.384370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a937e7d8-6979-45f9-bf70-1aa7b5a6678d" (UID: "a937e7d8-6979-45f9-bf70-1aa7b5a6678d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:27.385749 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.385728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a937e7d8-6979-45f9-bf70-1aa7b5a6678d" (UID: "a937e7d8-6979-45f9-bf70-1aa7b5a6678d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:27.386050 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.386033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7" (OuterVolumeSpecName: "kube-api-access-4s8d7") pod "a937e7d8-6979-45f9-bf70-1aa7b5a6678d" (UID: "a937e7d8-6979-45f9-bf70-1aa7b5a6678d"). InnerVolumeSpecName "kube-api-access-4s8d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:27.458890 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.458854    2581 generic.go:356] "Generic (PLEG): container finished" podID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerID="9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40" exitCode=2
Sep 16 19:27:27.458890 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.458884    2581 generic.go:356] "Generic (PLEG): container finished" podID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerID="f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79" exitCode=0
Sep 16 19:27:27.459066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.458911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"
Sep 16 19:27:27.459066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.458940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerDied","Data":"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"}
Sep 16 19:27:27.459066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.458997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerDied","Data":"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"}
Sep 16 19:27:27.459066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.459017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj" event={"ID":"a937e7d8-6979-45f9-bf70-1aa7b5a6678d","Type":"ContainerDied","Data":"18d99a0409dcdd675b220a0c3ec63cfc306190b3bbb80e338882e698bb864440"}
Sep 16 19:27:27.459066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.459044    2581 scope.go:117] "RemoveContainer" containerID="9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"
Sep 16 19:27:27.467255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.467238    2581 scope.go:117] "RemoveContainer" containerID="f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"
Sep 16 19:27:27.475000 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.474978    2581 scope.go:117] "RemoveContainer" containerID="ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"
Sep 16 19:27:27.476424 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.476402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:27.480271 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.480250    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj"]
Sep 16 19:27:27.484822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.484796    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:27.484822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.484819    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:27.484969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.484830    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:27.484969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.484840    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:27.484969 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.484849    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8d7\" (UniqueName: \"kubernetes.io/projected/a937e7d8-6979-45f9-bf70-1aa7b5a6678d-kube-api-access-4s8d7\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:27:27.485401 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.485387    2581 scope.go:117] "RemoveContainer" containerID="9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"
Sep 16 19:27:27.485683 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:27.485659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40\": container with ID starting with 9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40 not found: ID does not exist" containerID="9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"
Sep 16 19:27:27.485768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.485684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"} err="failed to get container status \"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40\": rpc error: code = NotFound desc = could not find container \"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40\": container with ID starting with 9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40 not found: ID does not exist"
Sep 16 19:27:27.485768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.485700    2581 scope.go:117] "RemoveContainer" containerID="f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"
Sep 16 19:27:27.485988 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:27.485966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79\": container with ID starting with f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79 not found: ID does not exist" containerID="f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"
Sep 16 19:27:27.486066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.485991    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"} err="failed to get container status \"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79\": rpc error: code = NotFound desc = could not find container \"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79\": container with ID starting with f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79 not found: ID does not exist"
Sep 16 19:27:27.486066 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486008    2581 scope.go:117] "RemoveContainer" containerID="ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"
Sep 16 19:27:27.486313 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:27:27.486292    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466\": container with ID starting with ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466 not found: ID does not exist" containerID="ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"
Sep 16 19:27:27.486393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486316    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"} err="failed to get container status \"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466\": rpc error: code = NotFound desc = could not find container \"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466\": container with ID starting with ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466 not found: ID does not exist"
Sep 16 19:27:27.486393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486331    2581 scope.go:117] "RemoveContainer" containerID="9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"
Sep 16 19:27:27.486578 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486558    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40"} err="failed to get container status \"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40\": rpc error: code = NotFound desc = could not find container \"9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40\": container with ID starting with 9f409bb23edf745053720f27be2aae6027154051329a459e49c57e3e09c5de40 not found: ID does not exist"
Sep 16 19:27:27.486639 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486578    2581 scope.go:117] "RemoveContainer" containerID="f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"
Sep 16 19:27:27.486805 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486790    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79"} err="failed to get container status \"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79\": rpc error: code = NotFound desc = could not find container \"f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79\": container with ID starting with f98c17e3dc617b5e98bc3f2ca696f7e6a6594b3d3026a64ace9639bbfce92e79 not found: ID does not exist"
Sep 16 19:27:27.486866 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.486804    2581 scope.go:117] "RemoveContainer" containerID="ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"
Sep 16 19:27:27.487033 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:27.487013    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466"} err="failed to get container status \"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466\": rpc error: code = NotFound desc = could not find container \"ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466\": container with ID starting with ed65da1f476e5bb26a39c46e7c2eb84dde960523bd06833b307e5a8325078466 not found: ID does not exist"
Sep 16 19:27:29.414694 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:29.414650    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" path="/var/lib/kubelet/pods/a937e7d8-6979-45f9-bf70-1aa7b5a6678d/volumes"
Sep 16 19:27:35.432846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:35.432809    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:45.432814 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:45.432768    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:55.432238 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:27:55.432197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:28:05.432295 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:05.432254    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:28:15.432579 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:15.432541    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:28:25.432803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:25.432762    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:28:33.414860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:33.414828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:28:39.429911 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.429831    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:28:39.430370 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.430061    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" containerID="cri-o://aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848" gracePeriod=30
Sep 16 19:28:39.430370 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.430094    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kube-rbac-proxy" containerID="cri-o://f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0" gracePeriod=30
Sep 16 19:28:39.602529 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:28:39.602905 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602879    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kube-rbac-proxy"
Sep 16 19:28:39.602905 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602905    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kube-rbac-proxy"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602950    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602964    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="storage-initializer"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.602973    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="storage-initializer"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.603042    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kserve-container"
Sep 16 19:28:39.603089 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.603052    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a937e7d8-6979-45f9-bf70-1aa7b5a6678d" containerName="kube-rbac-proxy"
Sep 16 19:28:39.606071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.606052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.608398 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.608375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:39.608506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.608375    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 16 19:28:39.608506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.608463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:28:39.616113 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.616091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:28:39.700475 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.700388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pl5b\" (UniqueName: \"kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.700475 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.700426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.700657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.700505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.700657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.700570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.700729 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.700659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.789700 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.789667    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerID="f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0" exitCode=2
Sep 16 19:28:39.789867 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.789735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerDied","Data":"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"}
Sep 16 19:28:39.801086 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801199 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pl5b\" (UniqueName: \"kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801199 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801199 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801360 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801414 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:28:39.801383    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 16 19:28:39.801466 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:28:39.801438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls podName:f473096f-69aa-4f6d-b084-dd040c299c4e nodeName:}" failed. No retries permitted until 2026-09-16 19:28:40.301421221 +0000 UTC m=+2695.363760451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 16 19:28:39.801657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801881 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.801971 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.801902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:39.809991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:39.809973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pl5b\" (UniqueName: \"kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:40.306774 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:40.306732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:40.309263 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:40.309245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:40.426907 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:40.426865    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:28:40.516880 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:40.516845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:40.796177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:40.796134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:28:40.798750 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:28:40.798725    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf473096f_69aa_4f6d_b084_dd040c299c4e.slice/crio-56128b1a100bec7193390ba4ab227504c8232d57dcbbf4e4eb0d1f7bec075286 WatchSource:0}: Error finding container 56128b1a100bec7193390ba4ab227504c8232d57dcbbf4e4eb0d1f7bec075286: Status 404 returned error can't find the container with id 56128b1a100bec7193390ba4ab227504c8232d57dcbbf4e4eb0d1f7bec075286
Sep 16 19:28:41.800888 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:41.800856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerStarted","Data":"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"}
Sep 16 19:28:41.801282 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:41.800914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerStarted","Data":"56128b1a100bec7193390ba4ab227504c8232d57dcbbf4e4eb0d1f7bec075286"}
Sep 16 19:28:42.812077 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:42.812038    2581 generic.go:356] "Generic (PLEG): container finished" podID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerID="1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d" exitCode=0
Sep 16 19:28:42.812543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:42.812086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerDied","Data":"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"}
Sep 16 19:28:43.409874 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.409835    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:28:43.819292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.819208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerStarted","Data":"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"}
Sep 16 19:28:43.819292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.819263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerStarted","Data":"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"}
Sep 16 19:28:43.819717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.819438    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:43.819717 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.819497    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:43.820623 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.820599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:28:43.839834 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:43.839792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podStartSLOduration=3.815806594 podStartE2EDuration="4.839779408s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.023972814s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:43.837957082 +0000 UTC m=+2698.900296336" watchObservedRunningTime="2026-09-16 19:28:43.839779408 +0000 UTC m=+2698.902118659"
Sep 16 19:28:44.283651 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.283624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:28:44.341408 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config\") pod \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") "
Sep 16 19:28:44.341562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location\") pod \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") "
Sep 16 19:28:44.341562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls\") pod \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") "
Sep 16 19:28:44.341562 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5gxb\" (UniqueName: \"kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb\") pod \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\" (UID: \"aa2edc88-68ce-4ea9-a6c1-af195a87f525\") "
Sep 16 19:28:44.341777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa2edc88-68ce-4ea9-a6c1-af195a87f525" (UID: "aa2edc88-68ce-4ea9-a6c1-af195a87f525"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:44.341842 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.341788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa2edc88-68ce-4ea9-a6c1-af195a87f525" (UID: "aa2edc88-68ce-4ea9-a6c1-af195a87f525"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:44.343680 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.343633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa2edc88-68ce-4ea9-a6c1-af195a87f525" (UID: "aa2edc88-68ce-4ea9-a6c1-af195a87f525"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:44.343738 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.343693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb" (OuterVolumeSpecName: "kube-api-access-c5gxb") pod "aa2edc88-68ce-4ea9-a6c1-af195a87f525" (UID: "aa2edc88-68ce-4ea9-a6c1-af195a87f525"). InnerVolumeSpecName "kube-api-access-c5gxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:44.443137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.443095    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:28:44.443137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.443127    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:28:44.443137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.443138    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa2edc88-68ce-4ea9-a6c1-af195a87f525-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:28:44.443380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.443171    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gxb\" (UniqueName: \"kubernetes.io/projected/aa2edc88-68ce-4ea9-a6c1-af195a87f525-kube-api-access-c5gxb\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:28:44.826158 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826119    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerID="aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848" exitCode=0
Sep 16 19:28:44.826627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826204    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"
Sep 16 19:28:44.826627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerDied","Data":"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"}
Sep 16 19:28:44.826627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx" event={"ID":"aa2edc88-68ce-4ea9-a6c1-af195a87f525","Type":"ContainerDied","Data":"069ab23bf53b8b591b0d6babcacaa65326f52498306c3dabb8be43b8586ecf20"}
Sep 16 19:28:44.826627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826297    2581 scope.go:117] "RemoveContainer" containerID="f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"
Sep 16 19:28:44.826870 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.826799    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:28:44.835196 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.835136    2581 scope.go:117] "RemoveContainer" containerID="aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"
Sep 16 19:28:44.842341 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.842326    2581 scope.go:117] "RemoveContainer" containerID="7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"
Sep 16 19:28:44.848722 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.848702    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:28:44.849684 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.849671    2581 scope.go:117] "RemoveContainer" containerID="f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"
Sep 16 19:28:44.849955 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:28:44.849935    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0\": container with ID starting with f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0 not found: ID does not exist" containerID="f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"
Sep 16 19:28:44.850023 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.849964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0"} err="failed to get container status \"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0\": rpc error: code = NotFound desc = could not find container \"f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0\": container with ID starting with f9d7d6eef60efec992244c0ddbea139d7669f8c2f37d6f5a1b2e22cace0ee6f0 not found: ID does not exist"
Sep 16 19:28:44.850023 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.849981    2581 scope.go:117] "RemoveContainer" containerID="aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"
Sep 16 19:28:44.850252 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:28:44.850234    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848\": container with ID starting with aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848 not found: ID does not exist" containerID="aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"
Sep 16 19:28:44.850321 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.850255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848"} err="failed to get container status \"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848\": rpc error: code = NotFound desc = could not find container \"aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848\": container with ID starting with aa1a148bc294e0265fa308d906bfacce9f343e7ac8f43ca28efa3cc993a47848 not found: ID does not exist"
Sep 16 19:28:44.850321 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.850269    2581 scope.go:117] "RemoveContainer" containerID="7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"
Sep 16 19:28:44.850543 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:28:44.850519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637\": container with ID starting with 7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637 not found: ID does not exist" containerID="7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"
Sep 16 19:28:44.850596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.850549    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637"} err="failed to get container status \"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637\": rpc error: code = NotFound desc = could not find container \"7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637\": container with ID starting with 7e7e0d56deca03f9e5470f8a3ade9190301f8c40296c9dce62d169b3966c6637 not found: ID does not exist"
Sep 16 19:28:44.852575 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:44.852554    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx"]
Sep 16 19:28:45.415142 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:45.415110    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" path="/var/lib/kubelet/pods/aa2edc88-68ce-4ea9-a6c1-af195a87f525/volumes"
Sep 16 19:28:49.831255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:49.831225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:28:49.831815 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:49.831790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:28:59.831786 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:28:59.831748    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:09.832007 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:09.831967    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:19.832389 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:19.832347    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:29.831676 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:29.831632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:39.832718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:39.832675    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:49.832012 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:49.831969    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:52.249967 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.249926    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250289    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kube-rbac-proxy"
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250301    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kube-rbac-proxy"
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250310    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container"
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250315    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container"
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250331    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="storage-initializer"
Sep 16 19:29:52.250367 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="storage-initializer"
Sep 16 19:29:52.250565 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250391    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kserve-container"
Sep 16 19:29:52.250565 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.250400    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2edc88-68ce-4ea9-a6c1-af195a87f525" containerName="kube-rbac-proxy"
Sep 16 19:29:52.253513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.253495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.255758 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.255734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 16 19:29:52.255758 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.255764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:52.256005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.255985    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 16 19:29:52.264804 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.264779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:29:52.411530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.411491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.411695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.411562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82z8l\" (UniqueName: \"kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.411695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.411592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.411695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.411622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.411811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.411689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512337 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82z8l\" (UniqueName: \"kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512337 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512337 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512337 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.512754 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.513006 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.512989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.513080 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.513062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.514833 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.514811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.524382 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.524360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82z8l\" (UniqueName: \"kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q7648\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.564139 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.564112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:29:52.846261 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:52.846233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:29:52.851282 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:29:52.851258    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa3a157_59bf_4f9e_ae4d_ab9a816b71ba.slice/crio-59c8803e0fe9b7970132dbf37c02adae29f109d0117beaa82498ccfc532aed2d WatchSource:0}: Error finding container 59c8803e0fe9b7970132dbf37c02adae29f109d0117beaa82498ccfc532aed2d: Status 404 returned error can't find the container with id 59c8803e0fe9b7970132dbf37c02adae29f109d0117beaa82498ccfc532aed2d
Sep 16 19:29:53.130940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:53.130856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerStarted","Data":"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"}
Sep 16 19:29:53.130940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:53.130914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerStarted","Data":"59c8803e0fe9b7970132dbf37c02adae29f109d0117beaa82498ccfc532aed2d"}
Sep 16 19:29:55.411415 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:55.411359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:29:58.155202 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:58.155143    2581 generic.go:356] "Generic (PLEG): container finished" podID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerID="d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030" exitCode=0
Sep 16 19:29:58.155672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:58.155225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerDied","Data":"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"}
Sep 16 19:29:58.156554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:29:58.156537    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:30:05.416570 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:05.416535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:30:09.652660 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:09.652621    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:30:09.653033 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:09.652926    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container" containerID="cri-o://809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d" gracePeriod=30
Sep 16 19:30:09.653033 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:09.653003    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kube-rbac-proxy" containerID="cri-o://c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871" gracePeriod=30
Sep 16 19:30:09.827375 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:09.827336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 16 19:30:10.219361 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.219325    2581 generic.go:356] "Generic (PLEG): container finished" podID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerID="c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871" exitCode=2
Sep 16 19:30:10.219532 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.219395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerDied","Data":"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"}
Sep 16 19:30:10.221481 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.221452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerStarted","Data":"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"}
Sep 16 19:30:10.221723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.221508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerStarted","Data":"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"}
Sep 16 19:30:10.221723 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.221704    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:30:10.240247 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:10.240196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podStartSLOduration=2.279326675 podStartE2EDuration="18.240179031s" totalImagesPullingTime="11.469885723s" totalInitContainerRuntime="4.490966633s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:10.24008513 +0000 UTC m=+2785.302424382" watchObservedRunningTime="2026-09-16 19:30:10.240179031 +0000 UTC m=+2785.302518280"
Sep 16 19:30:11.225981 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:11.225952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:30:11.226899 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:11.226875    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:12.230449 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:12.230410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:14.498987 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.498963    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:30:14.601852 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.601830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pl5b\" (UniqueName: \"kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b\") pod \"f473096f-69aa-4f6d-b084-dd040c299c4e\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") "
Sep 16 19:30:14.602017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.601861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert\") pod \"f473096f-69aa-4f6d-b084-dd040c299c4e\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") "
Sep 16 19:30:14.602017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.601878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config\") pod \"f473096f-69aa-4f6d-b084-dd040c299c4e\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") "
Sep 16 19:30:14.602017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.601916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location\") pod \"f473096f-69aa-4f6d-b084-dd040c299c4e\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") "
Sep 16 19:30:14.602017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.602001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") pod \"f473096f-69aa-4f6d-b084-dd040c299c4e\" (UID: \"f473096f-69aa-4f6d-b084-dd040c299c4e\") "
Sep 16 19:30:14.602260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.602231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f473096f-69aa-4f6d-b084-dd040c299c4e" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:14.602324 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.602292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f473096f-69aa-4f6d-b084-dd040c299c4e" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:14.602593 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.602568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f473096f-69aa-4f6d-b084-dd040c299c4e" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:14.603964 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.603938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b" (OuterVolumeSpecName: "kube-api-access-6pl5b") pod "f473096f-69aa-4f6d-b084-dd040c299c4e" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e"). InnerVolumeSpecName "kube-api-access-6pl5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:14.604069 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.604044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f473096f-69aa-4f6d-b084-dd040c299c4e" (UID: "f473096f-69aa-4f6d-b084-dd040c299c4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:14.703286 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.703256    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f473096f-69aa-4f6d-b084-dd040c299c4e-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:30:14.703286 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.703283    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pl5b\" (UniqueName: \"kubernetes.io/projected/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-api-access-6pl5b\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:30:14.703286 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.703292    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:30:14.703493 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.703301    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f473096f-69aa-4f6d-b084-dd040c299c4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:30:14.703493 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:14.703309    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f473096f-69aa-4f6d-b084-dd040c299c4e-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:30:15.246358 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.246319    2581 generic.go:356] "Generic (PLEG): container finished" podID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerID="809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d" exitCode=0
Sep 16 19:30:15.246530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.246380    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"
Sep 16 19:30:15.246530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.246401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerDied","Data":"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"}
Sep 16 19:30:15.246530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.246488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44" event={"ID":"f473096f-69aa-4f6d-b084-dd040c299c4e","Type":"ContainerDied","Data":"56128b1a100bec7193390ba4ab227504c8232d57dcbbf4e4eb0d1f7bec075286"}
Sep 16 19:30:15.246530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.246517    2581 scope.go:117] "RemoveContainer" containerID="c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"
Sep 16 19:30:15.254965 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.254948    2581 scope.go:117] "RemoveContainer" containerID="809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"
Sep 16 19:30:15.262416 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.262395    2581 scope.go:117] "RemoveContainer" containerID="1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"
Sep 16 19:30:15.269666 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.269648    2581 scope.go:117] "RemoveContainer" containerID="c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"
Sep 16 19:30:15.269907 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:30:15.269891    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871\": container with ID starting with c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871 not found: ID does not exist" containerID="c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"
Sep 16 19:30:15.269965 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.269915    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871"} err="failed to get container status \"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871\": rpc error: code = NotFound desc = could not find container \"c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871\": container with ID starting with c920e12094a6321cdd17a0e8d366f1f66d9ccb274684c3af53a37e96b19bf871 not found: ID does not exist"
Sep 16 19:30:15.269965 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.269939    2581 scope.go:117] "RemoveContainer" containerID="809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"
Sep 16 19:30:15.270197 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:30:15.270177    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d\": container with ID starting with 809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d not found: ID does not exist" containerID="809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"
Sep 16 19:30:15.270257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.270206    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d"} err="failed to get container status \"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d\": rpc error: code = NotFound desc = could not find container \"809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d\": container with ID starting with 809a859af6cf67e8e5ebf641492ca607799295e22696c28a9032e8370796c82d not found: ID does not exist"
Sep 16 19:30:15.270257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.270223    2581 scope.go:117] "RemoveContainer" containerID="1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"
Sep 16 19:30:15.270529 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:30:15.270473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d\": container with ID starting with 1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d not found: ID does not exist" containerID="1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"
Sep 16 19:30:15.270662 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.270538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d"} err="failed to get container status \"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d\": rpc error: code = NotFound desc = could not find container \"1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d\": container with ID starting with 1351d20991b5ab49f597e03242f5895f2ab63ef102c5cd3c9d1fed6751b1fe0d not found: ID does not exist"
Sep 16 19:30:15.272663 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.272641    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:30:15.279255 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.279237    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44"]
Sep 16 19:30:15.414468 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:15.414438    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" path="/var/lib/kubelet/pods/f473096f-69aa-4f6d-b084-dd040c299c4e/volumes"
Sep 16 19:30:17.235190 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:17.235138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:30:17.235649 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:17.235625    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:21.883716 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.883679    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884050    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884061    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884071    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kube-rbac-proxy"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884076    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kube-rbac-proxy"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="storage-initializer"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884091    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="storage-initializer"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kube-rbac-proxy"
Sep 16 19:30:21.884284 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.884195    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f473096f-69aa-4f6d-b084-dd040c299c4e" containerName="kserve-container"
Sep 16 19:30:21.887542 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.887516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:21.890748 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.890730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 16 19:30:21.890867 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.890775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:30:21.890867 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.890792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:30:21.898638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.898615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:30:21.965041 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.965003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:21.965241 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.965118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:21.965241 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.965170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:21.965241 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.965206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9md\" (UniqueName: \"kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:21.965241 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:21.965232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9md\" (UniqueName: \"kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.066787 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.066768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.067084 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.067062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.067124 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.067062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.068770 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.068744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.073671 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.073650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9md\" (UniqueName: \"kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.199177 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.199056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:22.475005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:22.474786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:30:22.477620 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:30:22.477593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43582a2_15e6_4395_9ecc_5d59596b46de.slice/crio-133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a WatchSource:0}: Error finding container 133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a: Status 404 returned error can't find the container with id 133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a
Sep 16 19:30:23.286458 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:23.286412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerStarted","Data":"390aafeab8756e07e2dd19f07c55ce1a26fe79971d8fb7535a0a1a6faab331ea"}
Sep 16 19:30:23.286886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:23.286500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerStarted","Data":"133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a"}
Sep 16 19:30:24.293195 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:24.293144    2581 generic.go:356] "Generic (PLEG): container finished" podID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerID="390aafeab8756e07e2dd19f07c55ce1a26fe79971d8fb7535a0a1a6faab331ea" exitCode=0
Sep 16 19:30:24.293636 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:24.293231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerDied","Data":"390aafeab8756e07e2dd19f07c55ce1a26fe79971d8fb7535a0a1a6faab331ea"}
Sep 16 19:30:25.300348 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.300316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerStarted","Data":"268560be2170a4e9ec180bb5194d3a9b2d63320b7c7488e57f2f9b962370adaa"}
Sep 16 19:30:25.300826 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.300379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerStarted","Data":"1a998ccad87b9496794a74c9b155e77af348ffdb10e4c1134bd57ce5a1a990b8"}
Sep 16 19:30:25.300826 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.300407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:25.300826 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.300446    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:25.301768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.301740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:30:25.322779 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:25.322714    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podStartSLOduration=3.287199788 podStartE2EDuration="4.322701118s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.03550133s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:25.321867988 +0000 UTC m=+2800.384207238" watchObservedRunningTime="2026-09-16 19:30:25.322701118 +0000 UTC m=+2800.385040369"
Sep 16 19:30:26.306011 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:26.305975    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:30:27.236543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:27.236504    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:31.311942 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:31.311913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:30:31.312426 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:31.312396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:30:37.235866 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:37.235822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:41.312543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:41.312503    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:30:47.235781 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:47.235740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:30:51.312862 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:51.312822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:30:57.237031 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:30:57.236995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:31:01.312872 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:01.312825    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:31:03.423425 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:03.423389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:31:03.423845 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:03.423612    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container" containerID="cri-o://d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3" gracePeriod=30
Sep 16 19:31:03.423845 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:03.423639    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kube-rbac-proxy" containerID="cri-o://521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50" gracePeriod=30
Sep 16 19:31:04.482820 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:04.482787    2581 generic.go:356] "Generic (PLEG): container finished" podID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerID="521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50" exitCode=2
Sep 16 19:31:04.483225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:04.482858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerDied","Data":"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"}
Sep 16 19:31:06.360260 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.360233    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:31:06.450204 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.450089    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82z8l\" (UniqueName: \"kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l\") pod \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") "
Sep 16 19:31:06.452138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.452114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l" (OuterVolumeSpecName: "kube-api-access-82z8l") pod "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" (UID: "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba"). InnerVolumeSpecName "kube-api-access-82z8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:06.496345 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.496315    2581 generic.go:356] "Generic (PLEG): container finished" podID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerID="d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3" exitCode=0
Sep 16 19:31:06.496499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.496378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerDied","Data":"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"}
Sep 16 19:31:06.496499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.496442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648" event={"ID":"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba","Type":"ContainerDied","Data":"59c8803e0fe9b7970132dbf37c02adae29f109d0117beaa82498ccfc532aed2d"}
Sep 16 19:31:06.496499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.496384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"
Sep 16 19:31:06.496499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.496476    2581 scope.go:117] "RemoveContainer" containerID="521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"
Sep 16 19:31:06.504670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.504643    2581 scope.go:117] "RemoveContainer" containerID="d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"
Sep 16 19:31:06.511993 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.511977    2581 scope.go:117] "RemoveContainer" containerID="d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"
Sep 16 19:31:06.519342 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.519325    2581 scope.go:117] "RemoveContainer" containerID="521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"
Sep 16 19:31:06.519593 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:31:06.519574    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50\": container with ID starting with 521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50 not found: ID does not exist" containerID="521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"
Sep 16 19:31:06.519637 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.519605    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50"} err="failed to get container status \"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50\": rpc error: code = NotFound desc = could not find container \"521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50\": container with ID starting with 521b7f2ba79da06c166c890cd24a457951a02595c773dd9e1ab729bc4cd96f50 not found: ID does not exist"
Sep 16 19:31:06.519637 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.519624    2581 scope.go:117] "RemoveContainer" containerID="d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"
Sep 16 19:31:06.519847 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:31:06.519829    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3\": container with ID starting with d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3 not found: ID does not exist" containerID="d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"
Sep 16 19:31:06.519914 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.519855    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3"} err="failed to get container status \"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3\": rpc error: code = NotFound desc = could not find container \"d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3\": container with ID starting with d3547efe9b989d315e0f6d8c2c076c4e447805c7f57cbb2c11ba5eaa8e706ad3 not found: ID does not exist"
Sep 16 19:31:06.519914 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.519876    2581 scope.go:117] "RemoveContainer" containerID="d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"
Sep 16 19:31:06.520100 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:31:06.520085    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030\": container with ID starting with d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030 not found: ID does not exist" containerID="d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"
Sep 16 19:31:06.520138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.520103    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030"} err="failed to get container status \"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030\": rpc error: code = NotFound desc = could not find container \"d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030\": container with ID starting with d2014ddf67e5b460854271177ad43404a220089b62f107780653e2c744857030 not found: ID does not exist"
Sep 16 19:31:06.550909 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.550889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config\") pod \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") "
Sep 16 19:31:06.550975 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.550944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location\") pod \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") "
Sep 16 19:31:06.551016 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.550980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls\") pod \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") "
Sep 16 19:31:06.551016 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.551008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert\") pod \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\" (UID: \"9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba\") "
Sep 16 19:31:06.551239 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.551216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" (UID: "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:06.551358 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.551340    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82z8l\" (UniqueName: \"kubernetes.io/projected/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-api-access-82z8l\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.551410 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.551395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" (UID: "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:06.552938 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.552917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" (UID: "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:06.558740 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.558717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" (UID: "9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:06.652567 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.652535    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.652567 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.652566    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.652740 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.652577    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.652740 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.652585    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.818283 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.818251    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:31:06.822618 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:06.822591    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648"]
Sep 16 19:31:07.414580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:07.414544    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" path="/var/lib/kubelet/pods/9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba/volumes"
Sep 16 19:31:08.540977 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.540943    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541479    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="storage-initializer"
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541499    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="storage-initializer"
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541513    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container"
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541522    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container"
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kube-rbac-proxy"
Sep 16 19:31:08.541586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541565    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kube-rbac-proxy"
Sep 16 19:31:08.541937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541646    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kube-rbac-proxy"
Sep 16 19:31:08.541937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.541661    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa3a157-59bf-4f9e-ae4d-ab9a816b71ba" containerName="kserve-container"
Sep 16 19:31:08.544961 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.544937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.547576 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.547554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:31:08.547689 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.547581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 16 19:31:08.547788 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.547775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:31:08.554989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.554963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:31:08.566912 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.566887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.567047 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.566940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9mq\" (UniqueName: \"kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.567047 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.567024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.567132 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.567084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.567202 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.567126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668502 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668671 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668671 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668671 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668887 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9mq\" (UniqueName: \"kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.668994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.668969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.669323 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.669302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.669415 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.669355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.671185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.671169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.678098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.678079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9mq\" (UniqueName: \"kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq\") pod \"isvc-paddle-runtime-predictor-58b99fff76-g5mt6\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:08.856781 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:08.856746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:09.130432 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:09.130407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:31:09.132854 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:31:09.132827    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a7363_63fd_4aef_9d47_3542fc743b77.slice/crio-bdb541a251b5a65b6ceabc814896aef34c5b7a525bf81c8a66dc1d8a65f1852b WatchSource:0}: Error finding container bdb541a251b5a65b6ceabc814896aef34c5b7a525bf81c8a66dc1d8a65f1852b: Status 404 returned error can't find the container with id bdb541a251b5a65b6ceabc814896aef34c5b7a525bf81c8a66dc1d8a65f1852b
Sep 16 19:31:09.513949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:09.513860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerStarted","Data":"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"}
Sep 16 19:31:09.513949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:09.513926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerStarted","Data":"bdb541a251b5a65b6ceabc814896aef34c5b7a525bf81c8a66dc1d8a65f1852b"}
Sep 16 19:31:11.312888 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:11.312845    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:31:13.533669 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:13.533636    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerID="70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa" exitCode=0
Sep 16 19:31:13.534035 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:13.533709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerDied","Data":"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"}
Sep 16 19:31:14.540588 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:14.540550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerStarted","Data":"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"}
Sep 16 19:31:14.540960 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:14.540644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerStarted","Data":"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"}
Sep 16 19:31:14.540960 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:14.540837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:14.561906 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:14.561856    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podStartSLOduration=2.358851487 podStartE2EDuration="6.561841439s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.202989952s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:14.559719878 +0000 UTC m=+2849.622059141" watchObservedRunningTime="2026-09-16 19:31:14.561841439 +0000 UTC m=+2849.624180690"
Sep 16 19:31:15.545138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:15.545108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:15.546623 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:15.546591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:16.550020 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:16.549980    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:21.312642 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:21.312599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:31:21.554869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:21.554840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:31:21.555429 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:21.555405    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:31.312768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:31.312730    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:31:31.556262 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:31.556220    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:41.313361 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:41.313289    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:31:41.556056 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:41.556011    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:41.955285 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:41.955242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:31:41.955553 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:41.955508    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container" containerID="cri-o://1a998ccad87b9496794a74c9b155e77af348ffdb10e4c1134bd57ce5a1a990b8" gracePeriod=30
Sep 16 19:31:41.955670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:41.955578    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kube-rbac-proxy" containerID="cri-o://268560be2170a4e9ec180bb5194d3a9b2d63320b7c7488e57f2f9b962370adaa" gracePeriod=30
Sep 16 19:31:42.668672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:42.668640    2581 generic.go:356] "Generic (PLEG): container finished" podID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerID="268560be2170a4e9ec180bb5194d3a9b2d63320b7c7488e57f2f9b962370adaa" exitCode=2
Sep 16 19:31:42.669064 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:42.668719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerDied","Data":"268560be2170a4e9ec180bb5194d3a9b2d63320b7c7488e57f2f9b962370adaa"}
Sep 16 19:31:46.306645 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.306602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 16 19:31:46.690835 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.690804    2581 generic.go:356] "Generic (PLEG): container finished" podID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerID="1a998ccad87b9496794a74c9b155e77af348ffdb10e4c1134bd57ce5a1a990b8" exitCode=0
Sep 16 19:31:46.690989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.690846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerDied","Data":"1a998ccad87b9496794a74c9b155e77af348ffdb10e4c1134bd57ce5a1a990b8"}
Sep 16 19:31:46.690989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.690904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt" event={"ID":"a43582a2-15e6-4395-9ecc-5d59596b46de","Type":"ContainerDied","Data":"133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a"}
Sep 16 19:31:46.690989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.690925    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133a22b887eb93dec1539aad2388556211d29d48ba21e39623ab621672e2103a"
Sep 16 19:31:46.702228 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.702204    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:31:46.800560 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.800523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls\") pod \"a43582a2-15e6-4395-9ecc-5d59596b46de\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") "
Sep 16 19:31:46.800760 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.800569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config\") pod \"a43582a2-15e6-4395-9ecc-5d59596b46de\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") "
Sep 16 19:31:46.800760 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.800604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location\") pod \"a43582a2-15e6-4395-9ecc-5d59596b46de\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") "
Sep 16 19:31:46.800760 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.800627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert\") pod \"a43582a2-15e6-4395-9ecc-5d59596b46de\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") "
Sep 16 19:31:46.800760 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.800691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp9md\" (UniqueName: \"kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md\") pod \"a43582a2-15e6-4395-9ecc-5d59596b46de\" (UID: \"a43582a2-15e6-4395-9ecc-5d59596b46de\") "
Sep 16 19:31:46.801060 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.801004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a43582a2-15e6-4395-9ecc-5d59596b46de" (UID: "a43582a2-15e6-4395-9ecc-5d59596b46de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:46.801060 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.801001    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a43582a2-15e6-4395-9ecc-5d59596b46de" (UID: "a43582a2-15e6-4395-9ecc-5d59596b46de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:46.801361 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.801274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a43582a2-15e6-4395-9ecc-5d59596b46de" (UID: "a43582a2-15e6-4395-9ecc-5d59596b46de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:46.802921 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.802902    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a43582a2-15e6-4395-9ecc-5d59596b46de" (UID: "a43582a2-15e6-4395-9ecc-5d59596b46de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:46.803262 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.803240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md" (OuterVolumeSpecName: "kube-api-access-vp9md") pod "a43582a2-15e6-4395-9ecc-5d59596b46de" (UID: "a43582a2-15e6-4395-9ecc-5d59596b46de"). InnerVolumeSpecName "kube-api-access-vp9md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:46.901415 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.901388    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a43582a2-15e6-4395-9ecc-5d59596b46de-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:46.901415 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.901414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:46.901609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.901423    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a43582a2-15e6-4395-9ecc-5d59596b46de-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:46.901609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.901434    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a43582a2-15e6-4395-9ecc-5d59596b46de-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:46.901609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:46.901442    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp9md\" (UniqueName: \"kubernetes.io/projected/a43582a2-15e6-4395-9ecc-5d59596b46de-kube-api-access-vp9md\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:31:47.696380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:47.696299    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"
Sep 16 19:31:47.715491 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:47.715457    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:31:47.719532 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:47.719505    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt"]
Sep 16 19:31:49.414975 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:49.414943    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" path="/var/lib/kubelet/pods/a43582a2-15e6-4395-9ecc-5d59596b46de/volumes"
Sep 16 19:31:51.555461 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:51.555423    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:31:54.099268 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:31:54.099638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099608    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container"
Sep 16 19:31:54.099638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099624    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container"
Sep 16 19:31:54.099718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099646    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kube-rbac-proxy"
Sep 16 19:31:54.099718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099651    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kube-rbac-proxy"
Sep 16 19:31:54.099718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099660    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="storage-initializer"
Sep 16 19:31:54.099718 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099666    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="storage-initializer"
Sep 16 19:31:54.099844 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099721    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kube-rbac-proxy"
Sep 16 19:31:54.099844 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.099730    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43582a2-15e6-4395-9ecc-5d59596b46de" containerName="kserve-container"
Sep 16 19:31:54.104281 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.104261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.106846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.106818    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 19:31:54.106846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.106833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:31:54.107000 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.106901    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:31:54.117553 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.117523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:31:54.270809 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.270769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.271005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.270838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.271005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.270873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.271005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.270919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.271005 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.270986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddwv\" (UniqueName: \"kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.371978 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.371895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.371978 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.371952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddwv\" (UniqueName: \"kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372215 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.371982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372215 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.372020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372328 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.372213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372525 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.372502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372803 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.372783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.372861 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.372818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.374408 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.374391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.381470 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.381424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddwv\" (UniqueName: \"kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.415692 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.415668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:54.686786 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.686761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:31:54.690040 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:31:54.690011    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d92ba3_d3eb_4b5b_b232_944ae7236e2d.slice/crio-5758f32119cdc27149f61bed9bb52f8ae09629e62d1c375f8d173a1e926d437a WatchSource:0}: Error finding container 5758f32119cdc27149f61bed9bb52f8ae09629e62d1c375f8d173a1e926d437a: Status 404 returned error can't find the container with id 5758f32119cdc27149f61bed9bb52f8ae09629e62d1c375f8d173a1e926d437a
Sep 16 19:31:54.731625 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:54.731600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerStarted","Data":"5758f32119cdc27149f61bed9bb52f8ae09629e62d1c375f8d173a1e926d437a"}
Sep 16 19:31:55.739527 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:55.739479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerStarted","Data":"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"}
Sep 16 19:31:56.745433 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:56.745397    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerID="1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6" exitCode=0
Sep 16 19:31:56.745853 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:56.745490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerDied","Data":"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"}
Sep 16 19:31:57.752475 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:57.752438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerStarted","Data":"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"}
Sep 16 19:31:57.752861 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:57.752505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerStarted","Data":"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"}
Sep 16 19:31:57.752861 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:57.752583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:57.774798 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:57.774747    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podStartSLOduration=2.761147257 podStartE2EDuration="3.774730465s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.013583208s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:57.772517714 +0000 UTC m=+2892.834856978" watchObservedRunningTime="2026-09-16 19:31:57.774730465 +0000 UTC m=+2892.837069717"
Sep 16 19:31:58.757352 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:58.757318    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:31:58.758812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:58.758784    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:31:59.761962 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:31:59.761923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:01.556927 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:01.556900    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:32:04.766949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:04.766916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:32:04.767483 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:04.767398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:09.634070 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:09.634031    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:32:09.634494 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:09.634309    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" containerID="cri-o://f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951" gracePeriod=30
Sep 16 19:32:09.634574 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:09.634498    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kube-rbac-proxy" containerID="cri-o://450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796" gracePeriod=30
Sep 16 19:32:09.812880 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:09.812845    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerID="450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796" exitCode=2
Sep 16 19:32:09.813045 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:09.812894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerDied","Data":"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"}
Sep 16 19:32:11.551211 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:11.551173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 16 19:32:11.555325 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:11.555292    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:32:12.379045 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.379022    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:32:12.407254 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert\") pod \"ad3a7363-63fd-4aef-9d47-3542fc743b77\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") "
Sep 16 19:32:12.407356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps9mq\" (UniqueName: \"kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq\") pod \"ad3a7363-63fd-4aef-9d47-3542fc743b77\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") "
Sep 16 19:32:12.407356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location\") pod \"ad3a7363-63fd-4aef-9d47-3542fc743b77\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") "
Sep 16 19:32:12.407356 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config\") pod \"ad3a7363-63fd-4aef-9d47-3542fc743b77\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") "
Sep 16 19:32:12.407514 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls\") pod \"ad3a7363-63fd-4aef-9d47-3542fc743b77\" (UID: \"ad3a7363-63fd-4aef-9d47-3542fc743b77\") "
Sep 16 19:32:12.407668 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad3a7363-63fd-4aef-9d47-3542fc743b77" (UID: "ad3a7363-63fd-4aef-9d47-3542fc743b77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:12.407757 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.407722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad3a7363-63fd-4aef-9d47-3542fc743b77" (UID: "ad3a7363-63fd-4aef-9d47-3542fc743b77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:12.409498 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.409477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq" (OuterVolumeSpecName: "kube-api-access-ps9mq") pod "ad3a7363-63fd-4aef-9d47-3542fc743b77" (UID: "ad3a7363-63fd-4aef-9d47-3542fc743b77"). InnerVolumeSpecName "kube-api-access-ps9mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:12.409586 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.409494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad3a7363-63fd-4aef-9d47-3542fc743b77" (UID: "ad3a7363-63fd-4aef-9d47-3542fc743b77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:12.420784 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.420759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad3a7363-63fd-4aef-9d47-3542fc743b77" (UID: "ad3a7363-63fd-4aef-9d47-3542fc743b77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:12.508630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.508543    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:32:12.508630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.508573    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9mq\" (UniqueName: \"kubernetes.io/projected/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-api-access-ps9mq\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:32:12.508630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.508582    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad3a7363-63fd-4aef-9d47-3542fc743b77-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:32:12.508630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.508592    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad3a7363-63fd-4aef-9d47-3542fc743b77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:32:12.508630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.508603    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad3a7363-63fd-4aef-9d47-3542fc743b77-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:32:12.828418 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.828325    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerID="f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951" exitCode=0
Sep 16 19:32:12.828846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.828440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"
Sep 16 19:32:12.828846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.828444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerDied","Data":"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"}
Sep 16 19:32:12.828846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.828593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6" event={"ID":"ad3a7363-63fd-4aef-9d47-3542fc743b77","Type":"ContainerDied","Data":"bdb541a251b5a65b6ceabc814896aef34c5b7a525bf81c8a66dc1d8a65f1852b"}
Sep 16 19:32:12.828846 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.828642    2581 scope.go:117] "RemoveContainer" containerID="450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"
Sep 16 19:32:12.837048 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.837029    2581 scope.go:117] "RemoveContainer" containerID="f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"
Sep 16 19:32:12.844568 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.844552    2581 scope.go:117] "RemoveContainer" containerID="70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"
Sep 16 19:32:12.850453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.850430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:32:12.852268 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.852227    2581 scope.go:117] "RemoveContainer" containerID="450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"
Sep 16 19:32:12.852524 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:32:12.852508    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796\": container with ID starting with 450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796 not found: ID does not exist" containerID="450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"
Sep 16 19:32:12.852590 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.852533    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796"} err="failed to get container status \"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796\": rpc error: code = NotFound desc = could not find container \"450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796\": container with ID starting with 450a81b4ed3c995370e0b1c56385ba731b009e695c2d2182979a908bf695a796 not found: ID does not exist"
Sep 16 19:32:12.852590 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.852549    2581 scope.go:117] "RemoveContainer" containerID="f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"
Sep 16 19:32:12.852789 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:32:12.852774    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951\": container with ID starting with f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951 not found: ID does not exist" containerID="f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"
Sep 16 19:32:12.852839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.852793    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951"} err="failed to get container status \"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951\": rpc error: code = NotFound desc = could not find container \"f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951\": container with ID starting with f68b7af2d89248650ef0ae765c14e1da4c1bb3d0c75229b98ddee01b3a224951 not found: ID does not exist"
Sep 16 19:32:12.852839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.852806    2581 scope.go:117] "RemoveContainer" containerID="70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"
Sep 16 19:32:12.853043 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:32:12.853025    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa\": container with ID starting with 70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa not found: ID does not exist" containerID="70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"
Sep 16 19:32:12.853091 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.853046    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa"} err="failed to get container status \"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa\": rpc error: code = NotFound desc = could not find container \"70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa\": container with ID starting with 70985063b6b95f92ecabe85810707ed1230f404c6e501a8c6d6106a8bd0226fa not found: ID does not exist"
Sep 16 19:32:12.860591 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:12.860571    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6"]
Sep 16 19:32:13.414761 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:13.414728    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" path="/var/lib/kubelet/pods/ad3a7363-63fd-4aef-9d47-3542fc743b77/volumes"
Sep 16 19:32:14.767374 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.767341    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:14.848536 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848502    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:32:14.848880 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848865    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="storage-initializer"
Sep 16 19:32:14.848928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848883    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="storage-initializer"
Sep 16 19:32:14.848928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container"
Sep 16 19:32:14.848928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container"
Sep 16 19:32:14.848928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848910    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kube-rbac-proxy"
Sep 16 19:32:14.848928 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848915    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kube-rbac-proxy"
Sep 16 19:32:14.849097 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848984    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kube-rbac-proxy"
Sep 16 19:32:14.849097 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.848997    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3a7363-63fd-4aef-9d47-3542fc743b77" containerName="kserve-container"
Sep 16 19:32:14.853749 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.853732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:14.856107 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.856086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 16 19:32:14.856243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.856128    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:32:14.856399 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.856384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:14.863609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.863584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:32:14.928912 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.928881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:14.929099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.928935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:14.929099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.928954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6x8w\" (UniqueName: \"kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:14.929099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.929008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:14.929099 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:14.929085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.029589 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.029494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.029589 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.029570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.029817 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.029597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6x8w\" (UniqueName: \"kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.029817 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.029630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.029817 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.029687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.030114 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.030093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.030408 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.030386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.030456 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.030385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.031996 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.031973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.042563 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.039305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6x8w\" (UniqueName: \"kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.165454 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.165410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:15.438098 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.438072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:32:15.440350 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:32:15.440319    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbd5d50_ab5a_430d_993c_b1ed84c66ea8.slice/crio-6c3ede7c76e479557a8741c363d3b4e7396116ae390515a91a87ebf2e30c6680 WatchSource:0}: Error finding container 6c3ede7c76e479557a8741c363d3b4e7396116ae390515a91a87ebf2e30c6680: Status 404 returned error can't find the container with id 6c3ede7c76e479557a8741c363d3b4e7396116ae390515a91a87ebf2e30c6680
Sep 16 19:32:15.844744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.844664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerStarted","Data":"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"}
Sep 16 19:32:15.844744 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:15.844722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerStarted","Data":"6c3ede7c76e479557a8741c363d3b4e7396116ae390515a91a87ebf2e30c6680"}
Sep 16 19:32:19.865061 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:19.865026    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerID="aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58" exitCode=0
Sep 16 19:32:19.865530 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:19.865110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerDied","Data":"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"}
Sep 16 19:32:20.871892 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:20.871861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerStarted","Data":"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"}
Sep 16 19:32:20.872290 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:20.871921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerStarted","Data":"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"}
Sep 16 19:32:20.872290 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:20.872113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:20.892000 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:20.891930    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podStartSLOduration=2.715223066 podStartE2EDuration="6.891910334s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.176687268s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:20.889846515 +0000 UTC m=+2915.952185774" watchObservedRunningTime="2026-09-16 19:32:20.891910334 +0000 UTC m=+2915.954249586"
Sep 16 19:32:21.876958 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:21.876923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:21.878474 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:21.878445    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:32:22.881873 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:22.881821    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:32:24.767667 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:24.767625    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:27.887313 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:27.887279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:32:27.887990 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:27.887955    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:32:34.767477 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:34.767432    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:37.888261 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:37.888210    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:32:44.767932 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:44.767887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:47.887952 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:47.887904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:32:54.767370 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:54.767332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:57.888386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:32:57.888344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:33:04.768256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:04.768212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:33:07.889358 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:07.889273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:33:14.767951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:14.767917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:33:16.224746 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:16.224710    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:33:16.225393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:16.224992    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" containerID="cri-o://47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e" gracePeriod=30
Sep 16 19:33:16.225393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:16.225027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kube-rbac-proxy" containerID="cri-o://1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467" gracePeriod=30
Sep 16 19:33:17.140550 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:17.140513    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerID="1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467" exitCode=2
Sep 16 19:33:17.140724 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:17.140563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerDied","Data":"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"}
Sep 16 19:33:17.882478 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:17.882435    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 16 19:33:17.888916 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:17.888882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:33:18.971493 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:18.971470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:33:19.072567 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6x8w\" (UniqueName: \"kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w\") pod \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") "
Sep 16 19:33:19.072567 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config\") pod \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") "
Sep 16 19:33:19.072567 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location\") pod \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") "
Sep 16 19:33:19.072791 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls\") pod \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") "
Sep 16 19:33:19.072791 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert\") pod \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\" (UID: \"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8\") "
Sep 16 19:33:19.073007 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.072987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" (UID: "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:19.073123 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.073101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" (UID: "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:19.074692 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.074663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" (UID: "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:19.074800 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.074705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w" (OuterVolumeSpecName: "kube-api-access-z6x8w") pod "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" (UID: "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8"). InnerVolumeSpecName "kube-api-access-z6x8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:19.078941 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.078921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" (UID: "4bbd5d50-ab5a-430d-993c-b1ed84c66ea8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:19.150453 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.150418    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerID="47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e" exitCode=0
Sep 16 19:33:19.150596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.150466    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"
Sep 16 19:33:19.150596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.150500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerDied","Data":"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"}
Sep 16 19:33:19.150596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.150554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv" event={"ID":"4bbd5d50-ab5a-430d-993c-b1ed84c66ea8","Type":"ContainerDied","Data":"6c3ede7c76e479557a8741c363d3b4e7396116ae390515a91a87ebf2e30c6680"}
Sep 16 19:33:19.150596 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.150579    2581 scope.go:117] "RemoveContainer" containerID="1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"
Sep 16 19:33:19.160032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.160015    2581 scope.go:117] "RemoveContainer" containerID="47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"
Sep 16 19:33:19.167510 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.167491    2581 scope.go:117] "RemoveContainer" containerID="aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"
Sep 16 19:33:19.173050 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173025    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:33:19.173366 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173348    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6x8w\" (UniqueName: \"kubernetes.io/projected/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-api-access-z6x8w\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:19.173428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173369    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:19.173428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173381    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:19.173428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173389    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:19.173428 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.173399    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:19.176247 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.176227    2581 scope.go:117] "RemoveContainer" containerID="1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"
Sep 16 19:33:19.176539 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:19.176516    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467\": container with ID starting with 1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467 not found: ID does not exist" containerID="1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"
Sep 16 19:33:19.176594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.176552    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467"} err="failed to get container status \"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467\": rpc error: code = NotFound desc = could not find container \"1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467\": container with ID starting with 1fc82f3715d6bd21017248c072ec7903f8a412deb9ca851c6b260aa96c412467 not found: ID does not exist"
Sep 16 19:33:19.176594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.176577    2581 scope.go:117] "RemoveContainer" containerID="47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"
Sep 16 19:33:19.176844 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:19.176824    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e\": container with ID starting with 47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e not found: ID does not exist" containerID="47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"
Sep 16 19:33:19.176899 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.176850    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e"} err="failed to get container status \"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e\": rpc error: code = NotFound desc = could not find container \"47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e\": container with ID starting with 47c0a61620b7fc099f82e4b21a5690b61a5df922c08382f0f16c61301019f77e not found: ID does not exist"
Sep 16 19:33:19.176899 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.176865    2581 scope.go:117] "RemoveContainer" containerID="aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"
Sep 16 19:33:19.177108 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:19.177090    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58\": container with ID starting with aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58 not found: ID does not exist" containerID="aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"
Sep 16 19:33:19.177172 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.177113    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58"} err="failed to get container status \"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58\": rpc error: code = NotFound desc = could not find container \"aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58\": container with ID starting with aa3394000854c3f16c9a6c9522c35608b32140f65d353bcb55f29a1e969d9f58 not found: ID does not exist"
Sep 16 19:33:19.177419 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.177403    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv"]
Sep 16 19:33:19.414445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:19.414414    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" path="/var/lib/kubelet/pods/4bbd5d50-ab5a-430d-993c-b1ed84c66ea8/volumes"
Sep 16 19:33:21.424003 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.423973    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424332    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container"
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container"
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424353    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kube-rbac-proxy"
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424359    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kube-rbac-proxy"
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424365    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="storage-initializer"
Sep 16 19:33:21.424381 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424371    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="storage-initializer"
Sep 16 19:33:21.424582 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424431    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kserve-container"
Sep 16 19:33:21.424582 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.424443    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bbd5d50-ab5a-430d-993c-b1ed84c66ea8" containerName="kube-rbac-proxy"
Sep 16 19:33:21.428865 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.428848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.431024 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.430985    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 16 19:33:21.431024 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.430996    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 16 19:33:21.431263 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.431249    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:33:21.438344 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.438320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:33:21.496393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.496364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbxk\" (UniqueName: \"kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.496555 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.496402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.496555 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.496486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.496661 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.496548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.496661 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.496598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.597763 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.597726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.597763 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.597765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.597992 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.597783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.597992 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.597816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbxk\" (UniqueName: \"kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.597992 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.597932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.598207 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.598186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.598493 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.598472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.598602 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.598552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.600386 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.600367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.606204 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.606179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbxk\" (UniqueName: \"kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk\") pod \"isvc-pmml-predictor-8674795f59-hcrpq\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:21.740334 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:21.740225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:22.013543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:22.013460    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:33:22.016507 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:33:22.016481    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd24b59c_5e16_4e93_b096_d47ccdfc32b4.slice/crio-de020764bb211bca846dacf5c6cbeca2fa332cec222e23258d873ad049417ad8 WatchSource:0}: Error finding container de020764bb211bca846dacf5c6cbeca2fa332cec222e23258d873ad049417ad8: Status 404 returned error can't find the container with id de020764bb211bca846dacf5c6cbeca2fa332cec222e23258d873ad049417ad8
Sep 16 19:33:22.166949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:22.166895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerStarted","Data":"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"}
Sep 16 19:33:22.167275 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:22.166978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerStarted","Data":"de020764bb211bca846dacf5c6cbeca2fa332cec222e23258d873ad049417ad8"}
Sep 16 19:33:24.127418 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:24.127388    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:33:24.127821 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:24.127613    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" containerID="cri-o://c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151" gracePeriod=30
Sep 16 19:33:24.127821 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:24.127653    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy" containerID="cri-o://5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5" gracePeriod=30
Sep 16 19:33:24.763221 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:24.763176    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 16 19:33:24.767446 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:24.767419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:33:25.185413 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.185376    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerID="5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5" exitCode=2
Sep 16 19:33:25.185772 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.185441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerDied","Data":"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"}
Sep 16 19:33:25.237014 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.236980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:25.240515 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.240499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.242721 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.242705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 19:33:25.242812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.242711    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:33:25.252049 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.252029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:25.327949 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.327919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.328062 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.327960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.328062 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.327977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.328062 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.327997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrmv\" (UniqueName: \"kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.429359 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.429359 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.429602 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.429602 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrmv\" (UniqueName: \"kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.429867 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.430006 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.429989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.431810 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.431792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.437842 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.437783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrmv\" (UniqueName: \"kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.550545 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.550511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:25.828112 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:25.828083    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:25.829568 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:33:25.829535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c27b5d3_01a9_4b9c_891f_5647949b515e.slice/crio-0b4372202e50ca586ca06d9b4863b3688f24215d22a0c6ce8536adedd1c95c3b WatchSource:0}: Error finding container 0b4372202e50ca586ca06d9b4863b3688f24215d22a0c6ce8536adedd1c95c3b: Status 404 returned error can't find the container with id 0b4372202e50ca586ca06d9b4863b3688f24215d22a0c6ce8536adedd1c95c3b
Sep 16 19:33:26.192654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:26.192612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerStarted","Data":"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"}
Sep 16 19:33:26.193123 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:26.192688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerStarted","Data":"0b4372202e50ca586ca06d9b4863b3688f24215d22a0c6ce8536adedd1c95c3b"}
Sep 16 19:33:26.194650 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:26.194620    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerID="9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6" exitCode=0
Sep 16 19:33:26.194769 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:26.194699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerDied","Data":"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"}
Sep 16 19:33:29.762534 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:29.762479    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 16 19:33:30.929771 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:30.929736    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:33:31.080777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.080742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location\") pod \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") "
Sep 16 19:33:31.080959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.080823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls\") pod \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") "
Sep 16 19:33:31.080959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.080875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert\") pod \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") "
Sep 16 19:33:31.080959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.080921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jddwv\" (UniqueName: \"kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv\") pod \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") "
Sep 16 19:33:31.080959 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.080950    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config\") pod \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\" (UID: \"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d\") "
Sep 16 19:33:31.081218 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.081087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" (UID: "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:31.081323 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.081305    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:31.081415 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.081314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" (UID: "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:31.081480 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.081413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" (UID: "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:31.083528 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.083494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv" (OuterVolumeSpecName: "kube-api-access-jddwv") pod "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" (UID: "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d"). InnerVolumeSpecName "kube-api-access-jddwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:31.083904 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.083876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" (UID: "f2d92ba3-d3eb-4b5b-b232-944ae7236e2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:31.182146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.182059    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:31.182146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.182093    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddwv\" (UniqueName: \"kubernetes.io/projected/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-api-access-jddwv\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:31.182146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.182110    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:31.182146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.182125    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:31.234276 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.234226    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerID="c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151" exitCode=0
Sep 16 19:33:31.234455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.234293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"
Sep 16 19:33:31.234455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.234307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerDied","Data":"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"}
Sep 16 19:33:31.234455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.234373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w" event={"ID":"f2d92ba3-d3eb-4b5b-b232-944ae7236e2d","Type":"ContainerDied","Data":"5758f32119cdc27149f61bed9bb52f8ae09629e62d1c375f8d173a1e926d437a"}
Sep 16 19:33:31.234455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.234409    2581 scope.go:117] "RemoveContainer" containerID="5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"
Sep 16 19:33:31.262435 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.262402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:33:31.268918 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.268890    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w"]
Sep 16 19:33:31.416627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:31.416592    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" path="/var/lib/kubelet/pods/f2d92ba3-d3eb-4b5b-b232-944ae7236e2d/volumes"
Sep 16 19:33:33.072551 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.072480    2581 scope.go:117] "RemoveContainer" containerID="c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"
Sep 16 19:33:33.213195 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.213103    2581 scope.go:117] "RemoveContainer" containerID="1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"
Sep 16 19:33:33.248128 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.248101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/0.log"
Sep 16 19:33:33.248310 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.248140    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerID="48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5" exitCode=1
Sep 16 19:33:33.248310 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.248182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerDied","Data":"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"}
Sep 16 19:33:33.262345 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.262320    2581 scope.go:117] "RemoveContainer" containerID="5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"
Sep 16 19:33:33.262635 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:33.262616    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5\": container with ID starting with 5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5 not found: ID does not exist" containerID="5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"
Sep 16 19:33:33.262699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.262644    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5"} err="failed to get container status \"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5\": rpc error: code = NotFound desc = could not find container \"5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5\": container with ID starting with 5630330af4adb29ff67691415cc505c12a7b59800f126e009183e9c17b6e3ad5 not found: ID does not exist"
Sep 16 19:33:33.262699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.262661    2581 scope.go:117] "RemoveContainer" containerID="c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"
Sep 16 19:33:33.262919 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:33.262901    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151\": container with ID starting with c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151 not found: ID does not exist" containerID="c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"
Sep 16 19:33:33.262960 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.262929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151"} err="failed to get container status \"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151\": rpc error: code = NotFound desc = could not find container \"c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151\": container with ID starting with c4bebed3c7691a1e3e0c6771444e986b06bf9430cf2ece42de6ed7edc3848151 not found: ID does not exist"
Sep 16 19:33:33.262960 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.262946    2581 scope.go:117] "RemoveContainer" containerID="1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"
Sep 16 19:33:33.263192 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:33.263178    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6\": container with ID starting with 1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6 not found: ID does not exist" containerID="1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"
Sep 16 19:33:33.263244 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:33.263195    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6"} err="failed to get container status \"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6\": rpc error: code = NotFound desc = could not find container \"1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6\": container with ID starting with 1ea6cc90655f8b8dc3ee6fe6de679e031434d4ae42a4f6cb2cec08b907a3b4e6 not found: ID does not exist"
Sep 16 19:33:34.256649 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:34.256622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/0.log"
Sep 16 19:33:34.257082 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:34.256740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerStarted","Data":"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"}
Sep 16 19:33:34.258962 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:34.258935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerStarted","Data":"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"}
Sep 16 19:33:34.259120 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:34.258996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerStarted","Data":"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"}
Sep 16 19:33:34.259210 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:34.259192    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:35.229734 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.229676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podStartSLOduration=3.299020548 podStartE2EDuration="14.229654701s" totalImagesPullingTime="7.148846588s" totalInitContainerRuntime="3.781787565s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:34.302212768 +0000 UTC m=+2989.364552019" watchObservedRunningTime="2026-09-16 19:33:35.229654701 +0000 UTC m=+2990.291993952"
Sep 16 19:33:35.230834 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.230815    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:35.264963 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.264924    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer" containerID="cri-o://783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7" gracePeriod=30
Sep 16 19:33:35.265443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.265343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:35.266318 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.266287    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:33:35.816633 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.816610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/1.log"
Sep 16 19:33:35.817039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.817023    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/0.log"
Sep 16 19:33:35.819432 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.819415    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:35.926951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.926865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls\") pod \"7c27b5d3-01a9-4b9c-891f-5647949b515e\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") "
Sep 16 19:33:35.926951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.926924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location\") pod \"7c27b5d3-01a9-4b9c-891f-5647949b515e\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") "
Sep 16 19:33:35.926951 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.926945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qrmv\" (UniqueName: \"kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv\") pod \"7c27b5d3-01a9-4b9c-891f-5647949b515e\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") "
Sep 16 19:33:35.927272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.926968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config\") pod \"7c27b5d3-01a9-4b9c-891f-5647949b515e\" (UID: \"7c27b5d3-01a9-4b9c-891f-5647949b515e\") "
Sep 16 19:33:35.927272 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.927215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c27b5d3-01a9-4b9c-891f-5647949b515e" (UID: "7c27b5d3-01a9-4b9c-891f-5647949b515e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:35.927409 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.927377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c27b5d3-01a9-4b9c-891f-5647949b515e" (UID: "7c27b5d3-01a9-4b9c-891f-5647949b515e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:35.929043 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.929023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv" (OuterVolumeSpecName: "kube-api-access-2qrmv") pod "7c27b5d3-01a9-4b9c-891f-5647949b515e" (UID: "7c27b5d3-01a9-4b9c-891f-5647949b515e"). InnerVolumeSpecName "kube-api-access-2qrmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:35.929111 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:35.929066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c27b5d3-01a9-4b9c-891f-5647949b515e" (UID: "7c27b5d3-01a9-4b9c-891f-5647949b515e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:36.027544 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.027511    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c27b5d3-01a9-4b9c-891f-5647949b515e-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:36.027544 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.027538    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c27b5d3-01a9-4b9c-891f-5647949b515e-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:36.027544 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.027549    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrmv\" (UniqueName: \"kubernetes.io/projected/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-api-access-2qrmv\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:36.027772 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.027557    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c27b5d3-01a9-4b9c-891f-5647949b515e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:33:36.271450 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271378    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/1.log"
Sep 16 19:33:36.271811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271770    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj_7c27b5d3-01a9-4b9c-891f-5647949b515e/storage-initializer/0.log"
Sep 16 19:33:36.271811 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271804    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerID="783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7" exitCode=1
Sep 16 19:33:36.271913 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerDied","Data":"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"}
Sep 16 19:33:36.271913 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271863    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"
Sep 16 19:33:36.272018 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.271952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj" event={"ID":"7c27b5d3-01a9-4b9c-891f-5647949b515e","Type":"ContainerDied","Data":"0b4372202e50ca586ca06d9b4863b3688f24215d22a0c6ce8536adedd1c95c3b"}
Sep 16 19:33:36.272018 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.272003    2581 scope.go:117] "RemoveContainer" containerID="783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"
Sep 16 19:33:36.272630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.272600    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:33:36.283840 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.283817    2581 scope.go:117] "RemoveContainer" containerID="48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"
Sep 16 19:33:36.293420 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.293399    2581 scope.go:117] "RemoveContainer" containerID="783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"
Sep 16 19:33:36.293795 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:36.293775    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7\": container with ID starting with 783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7 not found: ID does not exist" containerID="783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"
Sep 16 19:33:36.293878 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.293802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7"} err="failed to get container status \"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7\": rpc error: code = NotFound desc = could not find container \"783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7\": container with ID starting with 783e0e82195612009b35587c67b6aa044a776a7b0b4a6e31691c295bed27a4c7 not found: ID does not exist"
Sep 16 19:33:36.293878 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.293821    2581 scope.go:117] "RemoveContainer" containerID="48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"
Sep 16 19:33:36.294088 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:33:36.294075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5\": container with ID starting with 48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5 not found: ID does not exist" containerID="48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"
Sep 16 19:33:36.294129 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.294092    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5"} err="failed to get container status \"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5\": rpc error: code = NotFound desc = could not find container \"48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5\": container with ID starting with 48a5713ec1f680586d6a8574b3600c5cb14ae342e7cabc1e7b1fb85474b50ee5 not found: ID does not exist"
Sep 16 19:33:36.312263 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.312236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:36.316858 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.316836    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj"]
Sep 16 19:33:36.402699 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.402663    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:33:36.403209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403177    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy"
Sep 16 19:33:36.403209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403213    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403238    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403247    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403273    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403289    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403297    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="storage-initializer"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403306    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container"
Sep 16 19:33:36.403315 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403314    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container"
Sep 16 19:33:36.403558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403422    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kserve-container"
Sep 16 19:33:36.403558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403434    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d92ba3-d3eb-4b5b-b232-944ae7236e2d" containerName="kube-rbac-proxy"
Sep 16 19:33:36.403558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403442    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.403558 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.403552    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" containerName="storage-initializer"
Sep 16 19:33:36.407892 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.407877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.414839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.414819    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:33:36.414962 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.414904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 16 19:33:36.425084 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.425060    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:33:36.532350 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.532269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.532350 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.532317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.532537 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.532398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.532537 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.532448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.532537 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.532468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblf5\" (UniqueName: \"kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.632989 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.632943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633184 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633184 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblf5\" (UniqueName: \"kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633184 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633184 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633468 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633802 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.633909 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.633812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.635550 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.635534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.641609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.641592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblf5\" (UniqueName: \"kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5\") pod \"isvc-pmml-runtime-predictor-885fc8777-qz2qm\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:36.717580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:36.717543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:37.008546 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:37.008432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:33:37.011117 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:33:37.011090    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1687b1d_2d5b_4a06_9698_01bc53db56ec.slice/crio-b309d0b12f312fb8165387c48a38929181274f84db309c71ca2dfc6bd6a56a25 WatchSource:0}: Error finding container b309d0b12f312fb8165387c48a38929181274f84db309c71ca2dfc6bd6a56a25: Status 404 returned error can't find the container with id b309d0b12f312fb8165387c48a38929181274f84db309c71ca2dfc6bd6a56a25
Sep 16 19:33:37.279460 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:37.279361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerStarted","Data":"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"}
Sep 16 19:33:37.279460 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:37.279434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerStarted","Data":"b309d0b12f312fb8165387c48a38929181274f84db309c71ca2dfc6bd6a56a25"}
Sep 16 19:33:37.415056 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:37.415021    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c27b5d3-01a9-4b9c-891f-5647949b515e" path="/var/lib/kubelet/pods/7c27b5d3-01a9-4b9c-891f-5647949b515e/volumes"
Sep 16 19:33:41.276570 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:41.276536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:33:41.277139 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:41.277111    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:33:41.301876 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:41.301840    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerID="6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345" exitCode=0
Sep 16 19:33:41.302064 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:41.301915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerDied","Data":"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"}
Sep 16 19:33:42.309612 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:42.309574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerStarted","Data":"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"}
Sep 16 19:33:42.310013 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:42.309633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerStarted","Data":"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"}
Sep 16 19:33:42.310013 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:42.309899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:42.335644 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:42.335590    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podStartSLOduration=2.718416362 podStartE2EDuration="6.335573762s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.6171574s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:42.333047962 +0000 UTC m=+2997.395387212" watchObservedRunningTime="2026-09-16 19:33:42.335573762 +0000 UTC m=+2997.397913014"
Sep 16 19:33:43.314918 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:43.314884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:43.315999 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:43.315974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:33:44.318930 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:44.318888    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:33:49.323839 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:49.323805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:33:49.324418 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:49.324309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:33:51.277385 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:51.277347    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:33:59.324905 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:33:59.324864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:01.277238 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:01.277199    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:09.325246 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:09.325204    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:11.277409 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:11.277367    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:19.324730 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:19.324680    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:21.277583 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:21.277546    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:29.324943 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:29.324899    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:31.277406 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:31.277364    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:39.324884 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:39.324803    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:41.277127 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:41.277086    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:49.325172 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:49.325117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:34:51.277082 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:51.277043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:34:56.410163 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:56.410117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:34:59.324786 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:34:59.324740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:35:02.618938 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:02.618906    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:35:02.619388 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:02.619208    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container" containerID="cri-o://fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622" gracePeriod=30
Sep 16 19:35:02.619388 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:02.619208    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kube-rbac-proxy" containerID="cri-o://f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670" gracePeriod=30
Sep 16 19:35:03.688738 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:03.688703    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerID="f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670" exitCode=2
Sep 16 19:35:03.689128 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:03.688775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerDied","Data":"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"}
Sep 16 19:35:06.169335 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.169311    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:35:06.281072 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.280989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls\") pod \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") "
Sep 16 19:35:06.281072 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmbxk\" (UniqueName: \"kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk\") pod \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") "
Sep 16 19:35:06.281072 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert\") pod \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") "
Sep 16 19:35:06.281072 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config\") pod \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") "
Sep 16 19:35:06.281444 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location\") pod \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\" (UID: \"cd24b59c-5e16-4e93-b096-d47ccdfc32b4\") "
Sep 16 19:35:06.281444 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281412    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd24b59c-5e16-4e93-b096-d47ccdfc32b4" (UID: "cd24b59c-5e16-4e93-b096-d47ccdfc32b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:06.281444 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd24b59c-5e16-4e93-b096-d47ccdfc32b4" (UID: "cd24b59c-5e16-4e93-b096-d47ccdfc32b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:06.281556 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.281467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd24b59c-5e16-4e93-b096-d47ccdfc32b4" (UID: "cd24b59c-5e16-4e93-b096-d47ccdfc32b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:06.283143 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.283122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd24b59c-5e16-4e93-b096-d47ccdfc32b4" (UID: "cd24b59c-5e16-4e93-b096-d47ccdfc32b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:06.283243 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.283193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk" (OuterVolumeSpecName: "kube-api-access-cmbxk") pod "cd24b59c-5e16-4e93-b096-d47ccdfc32b4" (UID: "cd24b59c-5e16-4e93-b096-d47ccdfc32b4"). InnerVolumeSpecName "kube-api-access-cmbxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:06.381885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.381851    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:06.381885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.381880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbxk\" (UniqueName: \"kubernetes.io/projected/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-api-access-cmbxk\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:06.381885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.381891    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:06.382119 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.381901    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:06.382119 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.381909    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd24b59c-5e16-4e93-b096-d47ccdfc32b4-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:06.704376 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.704341    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerID="fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622" exitCode=0
Sep 16 19:35:06.704543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.704399    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"
Sep 16 19:35:06.704543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.704430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerDied","Data":"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"}
Sep 16 19:35:06.704543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.704490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq" event={"ID":"cd24b59c-5e16-4e93-b096-d47ccdfc32b4","Type":"ContainerDied","Data":"de020764bb211bca846dacf5c6cbeca2fa332cec222e23258d873ad049417ad8"}
Sep 16 19:35:06.704543 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.704516    2581 scope.go:117] "RemoveContainer" containerID="f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"
Sep 16 19:35:06.713042 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.713025    2581 scope.go:117] "RemoveContainer" containerID="fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"
Sep 16 19:35:06.720624 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.720604    2581 scope.go:117] "RemoveContainer" containerID="9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"
Sep 16 19:35:06.725444 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.725424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:35:06.729217 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.729191    2581 scope.go:117] "RemoveContainer" containerID="f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"
Sep 16 19:35:06.729547 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:06.729527    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670\": container with ID starting with f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670 not found: ID does not exist" containerID="f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"
Sep 16 19:35:06.729677 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.729554    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670"} err="failed to get container status \"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670\": rpc error: code = NotFound desc = could not find container \"f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670\": container with ID starting with f2822699924fd4f2407bc471d9d7d6007d78373f5adf3e618498d52ebb019670 not found: ID does not exist"
Sep 16 19:35:06.729677 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.729578    2581 scope.go:117] "RemoveContainer" containerID="fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"
Sep 16 19:35:06.729840 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:06.729811    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622\": container with ID starting with fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622 not found: ID does not exist" containerID="fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"
Sep 16 19:35:06.729944 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.729845    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622"} err="failed to get container status \"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622\": rpc error: code = NotFound desc = could not find container \"fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622\": container with ID starting with fb90f98b016b71675a521d1a0d12a1db358725284b1c757d0c93384ffb080622 not found: ID does not exist"
Sep 16 19:35:06.729944 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.729868    2581 scope.go:117] "RemoveContainer" containerID="9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"
Sep 16 19:35:06.730202 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:06.730180    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6\": container with ID starting with 9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6 not found: ID does not exist" containerID="9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"
Sep 16 19:35:06.730280 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.730208    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6"} err="failed to get container status \"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6\": rpc error: code = NotFound desc = could not find container \"9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6\": container with ID starting with 9b43102c09bb7f82560e041e48908a09b5fbdd6ee90efa6c9c580bcf5b4d8ce6 not found: ID does not exist"
Sep 16 19:35:06.730280 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:06.730266    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq"]
Sep 16 19:35:07.415390 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.415348    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" path="/var/lib/kubelet/pods/cd24b59c-5e16-4e93-b096-d47ccdfc32b4/volumes"
Sep 16 19:35:07.829865 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.829784    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:35:07.830192 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830177    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kube-rbac-proxy"
Sep 16 19:35:07.830192 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830192    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kube-rbac-proxy"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830206    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830211    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830225    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="storage-initializer"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830232    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="storage-initializer"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830292    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kube-rbac-proxy"
Sep 16 19:35:07.830302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.830303    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd24b59c-5e16-4e93-b096-d47ccdfc32b4" containerName="kserve-container"
Sep 16 19:35:07.834929 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.834914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.836902 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.836882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-404a29-predictor-serving-cert\""
Sep 16 19:35:07.837034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.836910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-404a29-kube-rbac-proxy-sar-config\""
Sep 16 19:35:07.842522 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.842181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:35:07.895274 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.895241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.895445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.895287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhzp\" (UniqueName: \"kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.895445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.895314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.895445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.895400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.895445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.895421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.996627 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.996597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.996785 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.996641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhzp\" (UniqueName: \"kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.996785 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.996671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.996785 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.996730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.996785 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.996750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.997232 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.997201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.997519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.997497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.997571 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.997497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:07.999318 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:07.999298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:08.004368 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:08.004348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhzp\" (UniqueName: \"kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp\") pod \"isvc-primary-404a29-predictor-8574c4b8bf-kvtlt\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") " pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:08.146480 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:08.146439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:08.417514 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:08.417467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:35:08.419695 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:35:08.419667    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3ee073_7d23_49f0_840c_4d97cdf9731b.slice/crio-1a86ee7c04d5a95415c4cb20433115251002e2ff9c3a23481d052a50a1f43619 WatchSource:0}: Error finding container 1a86ee7c04d5a95415c4cb20433115251002e2ff9c3a23481d052a50a1f43619: Status 404 returned error can't find the container with id 1a86ee7c04d5a95415c4cb20433115251002e2ff9c3a23481d052a50a1f43619
Sep 16 19:35:08.719437 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:08.719339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerStarted","Data":"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"}
Sep 16 19:35:08.719437 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:08.719410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerStarted","Data":"1a86ee7c04d5a95415c4cb20433115251002e2ff9c3a23481d052a50a1f43619"}
Sep 16 19:35:09.325456 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:09.325424    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:35:12.738257 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:12.738225    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerID="8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21" exitCode=0
Sep 16 19:35:12.738605 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:12.738301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerDied","Data":"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"}
Sep 16 19:35:13.745233 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.745192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerStarted","Data":"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"}
Sep 16 19:35:13.745630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.745272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerStarted","Data":"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"}
Sep 16 19:35:13.745630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.745571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:13.745630 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.745625    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:13.747053 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.747025    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:13.766782 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:13.766735    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podStartSLOduration=2.960150488 podStartE2EDuration="6.76671993s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806569442s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:13.764188977 +0000 UTC m=+3088.826528228" watchObservedRunningTime="2026-09-16 19:35:13.76671993 +0000 UTC m=+3088.829059180"
Sep 16 19:35:14.749987 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:14.749946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:17.426229 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:17.426197    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:35:17.426654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:17.426396    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" containerID="cri-o://fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa" gracePeriod=30
Sep 16 19:35:17.426654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:17.426440    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kube-rbac-proxy" containerID="cri-o://eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830" gracePeriod=30
Sep 16 19:35:17.764146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:17.763986    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerID="eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830" exitCode=2
Sep 16 19:35:17.764146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:17.764053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerDied","Data":"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"}
Sep 16 19:35:19.320215 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:19.320138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 16 19:35:19.324479 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:19.324451    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:35:19.755021 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:19.754992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:35:19.755555 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:19.755525    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:21.177282 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.177252    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:35:21.311347 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert\") pod \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") "
Sep 16 19:35:21.311347 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cblf5\" (UniqueName: \"kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5\") pod \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") "
Sep 16 19:35:21.311594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls\") pod \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") "
Sep 16 19:35:21.311594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config\") pod \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") "
Sep 16 19:35:21.311594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location\") pod \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\" (UID: \"b1687b1d-2d5b-4a06-9698-01bc53db56ec\") "
Sep 16 19:35:21.311820 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1687b1d-2d5b-4a06-9698-01bc53db56ec" (UID: "b1687b1d-2d5b-4a06-9698-01bc53db56ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:21.311935 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1687b1d-2d5b-4a06-9698-01bc53db56ec" (UID: "b1687b1d-2d5b-4a06-9698-01bc53db56ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:21.311935 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.311870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1687b1d-2d5b-4a06-9698-01bc53db56ec" (UID: "b1687b1d-2d5b-4a06-9698-01bc53db56ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:21.313580 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.313558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5" (OuterVolumeSpecName: "kube-api-access-cblf5") pod "b1687b1d-2d5b-4a06-9698-01bc53db56ec" (UID: "b1687b1d-2d5b-4a06-9698-01bc53db56ec"). InnerVolumeSpecName "kube-api-access-cblf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:21.313670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.313559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1687b1d-2d5b-4a06-9698-01bc53db56ec" (UID: "b1687b1d-2d5b-4a06-9698-01bc53db56ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:21.412185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.412139    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1687b1d-2d5b-4a06-9698-01bc53db56ec-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.412185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.412180    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.412185 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.412190    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.412474 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.412199    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1687b1d-2d5b-4a06-9698-01bc53db56ec-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.412474 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.412209    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblf5\" (UniqueName: \"kubernetes.io/projected/b1687b1d-2d5b-4a06-9698-01bc53db56ec-kube-api-access-cblf5\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:35:21.784209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.784171    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerID="fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa" exitCode=0
Sep 16 19:35:21.784384 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.784219    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"
Sep 16 19:35:21.784384 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.784246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerDied","Data":"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"}
Sep 16 19:35:21.784384 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.784313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm" event={"ID":"b1687b1d-2d5b-4a06-9698-01bc53db56ec","Type":"ContainerDied","Data":"b309d0b12f312fb8165387c48a38929181274f84db309c71ca2dfc6bd6a56a25"}
Sep 16 19:35:21.784384 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.784344    2581 scope.go:117] "RemoveContainer" containerID="eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"
Sep 16 19:35:21.796083 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.796061    2581 scope.go:117] "RemoveContainer" containerID="fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"
Sep 16 19:35:21.803015 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.802984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:35:21.803833 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.803819    2581 scope.go:117] "RemoveContainer" containerID="6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"
Sep 16 19:35:21.807639 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.807617    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm"]
Sep 16 19:35:21.811822 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.811806    2581 scope.go:117] "RemoveContainer" containerID="eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"
Sep 16 19:35:21.812060 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:21.812040    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830\": container with ID starting with eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830 not found: ID does not exist" containerID="eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"
Sep 16 19:35:21.812109 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.812068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830"} err="failed to get container status \"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830\": rpc error: code = NotFound desc = could not find container \"eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830\": container with ID starting with eaee5d1e103cbb0f53a32162921cf440dc84e4ea8b1a7149305184c75dd97830 not found: ID does not exist"
Sep 16 19:35:21.812109 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.812084    2581 scope.go:117] "RemoveContainer" containerID="fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"
Sep 16 19:35:21.812348 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:21.812329    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa\": container with ID starting with fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa not found: ID does not exist" containerID="fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"
Sep 16 19:35:21.812398 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.812355    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa"} err="failed to get container status \"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa\": rpc error: code = NotFound desc = could not find container \"fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa\": container with ID starting with fefa2a47cbd1ce1dd5040d6a7dc4a9ef6ff5a5a5176cad20e3bf80b7b1cbdbaa not found: ID does not exist"
Sep 16 19:35:21.812398 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.812371    2581 scope.go:117] "RemoveContainer" containerID="6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"
Sep 16 19:35:21.812617 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:35:21.812602    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345\": container with ID starting with 6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345 not found: ID does not exist" containerID="6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"
Sep 16 19:35:21.812657 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:21.812621    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345"} err="failed to get container status \"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345\": rpc error: code = NotFound desc = could not find container \"6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345\": container with ID starting with 6f5b2da193b8bdcf6fb29d0bf0903b18ea93024221bc805b335656182104f345 not found: ID does not exist"
Sep 16 19:35:22.617136 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="storage-initializer"
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617465    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="storage-initializer"
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container"
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617480    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container"
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kube-rbac-proxy"
Sep 16 19:35:22.617513 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617491    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kube-rbac-proxy"
Sep 16 19:35:22.617711 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617555    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kube-rbac-proxy"
Sep 16 19:35:22.617711 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.617564    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" containerName="kserve-container"
Sep 16 19:35:22.622296 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.622277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.624568 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.624544    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 16 19:35:22.624689 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.624551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:35:22.630532 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.630496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:35:22.724219 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.724180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwh66\" (UniqueName: \"kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.724416 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.724227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.724416 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.724329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.724416 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.724380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.724552 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.724442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.824963 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.824927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.824963 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.824964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825266 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.824986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwh66\" (UniqueName: \"kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825266 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.825008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825266 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.825061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825448 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.825424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.825790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.825860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.825790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.827422 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.827399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.833202 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.833178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwh66\" (UniqueName: \"kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:22.933146 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:22.933067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:23.207581 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:23.207513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:35:23.210508 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:35:23.210478    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0099380c_9fc0_4b3a_a992_60fdf238b39f.slice/crio-648660cbd05276dddf42ebbc2ea048ff2071cdd7cd608f04ba309180070680d2 WatchSource:0}: Error finding container 648660cbd05276dddf42ebbc2ea048ff2071cdd7cd608f04ba309180070680d2: Status 404 returned error can't find the container with id 648660cbd05276dddf42ebbc2ea048ff2071cdd7cd608f04ba309180070680d2
Sep 16 19:35:23.415733 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:23.415703    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1687b1d-2d5b-4a06-9698-01bc53db56ec" path="/var/lib/kubelet/pods/b1687b1d-2d5b-4a06-9698-01bc53db56ec/volumes"
Sep 16 19:35:23.797685 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:23.797644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerStarted","Data":"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"}
Sep 16 19:35:23.798095 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:23.797730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerStarted","Data":"648660cbd05276dddf42ebbc2ea048ff2071cdd7cd608f04ba309180070680d2"}
Sep 16 19:35:27.816632 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:27.816593    2581 generic.go:356] "Generic (PLEG): container finished" podID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerID="3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c" exitCode=0
Sep 16 19:35:27.817271 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:27.816640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerDied","Data":"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"}
Sep 16 19:35:28.823537 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.823501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerStarted","Data":"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"}
Sep 16 19:35:28.823920 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.823568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerStarted","Data":"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"}
Sep 16 19:35:28.823920 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.823906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:28.823994 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.823959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:28.825342 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.825314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:35:28.844201 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:28.844134    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podStartSLOduration=3.097612487 podStartE2EDuration="6.844121216s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746508729s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:28.842784489 +0000 UTC m=+3103.905123740" watchObservedRunningTime="2026-09-16 19:35:28.844121216 +0000 UTC m=+3103.906460467"
Sep 16 19:35:29.755452 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:29.755405    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:29.828232 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:29.828192    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:35:34.833555 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:34.833522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:35:34.834205 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:34.834179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:35:39.756352 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:39.756313    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:44.834598 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:44.834554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:35:49.755906 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:49.755865    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:35:54.835058 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:54.835017    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:35:59.756326 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:35:59.756280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:36:04.834135 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:04.834093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:36:09.755708 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:09.755622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:36:14.834789 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:14.834739    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:36:19.756312 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:19.756275    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:36:24.834956 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:24.834913    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:36:27.414709 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:27.414678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:36:34.834312 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:34.834268    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:36:43.003937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.003898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:43.007560 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.007537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.009886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.009863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-404a29-predictor-serving-cert\""
Sep 16 19:36:43.010071 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.010058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-404a29-dockercfg-nf4gq\""
Sep 16 19:36:43.010166 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.010079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-404a29\""
Sep 16 19:36:43.010302 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.010284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-404a29-kube-rbac-proxy-sar-config\""
Sep 16 19:36:43.018409 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.018388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:43.086464 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.086429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.086464 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.086465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.086670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.086490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.086670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.086615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsw2\" (UniqueName: \"kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.086670 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.086650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.187894 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.187850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsw2\" (UniqueName: \"kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.187894 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.187897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188141 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.187978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188141 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.188002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188141 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.188042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188325 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:43.188194    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-404a29-predictor-serving-cert: secret "isvc-secondary-404a29-predictor-serving-cert" not found
Sep 16 19:36:43.188325 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:43.188263    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls podName:480acdec-db55-405b-88c6-ab5bab2a3c6f nodeName:}" failed. No retries permitted until 2026-09-16 19:36:43.688243406 +0000 UTC m=+3178.750582640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls") pod "isvc-secondary-404a29-predictor-544c4b6766-nlpb8" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f") : secret "isvc-secondary-404a29-predictor-serving-cert" not found
Sep 16 19:36:43.188449 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.188389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188704 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.188681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.188836 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.188815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.196519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.196498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsw2\" (UniqueName: \"kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.691677 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.691638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.694105 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.694082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") pod \"isvc-secondary-404a29-predictor-544c4b6766-nlpb8\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") " pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:43.919172 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:43.919116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:44.201390 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:44.201360    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:44.203723 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:36:44.203686    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480acdec_db55_405b_88c6_ab5bab2a3c6f.slice/crio-9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7 WatchSource:0}: Error finding container 9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7: Status 404 returned error can't find the container with id 9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7
Sep 16 19:36:44.834451 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:44.834411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:36:45.177843 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:45.177806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerStarted","Data":"d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"}
Sep 16 19:36:45.178084 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:45.177874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerStarted","Data":"9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7"}
Sep 16 19:36:47.188745 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:47.188721    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/0.log"
Sep 16 19:36:47.189059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:47.188756    2581 generic.go:356] "Generic (PLEG): container finished" podID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerID="d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5" exitCode=1
Sep 16 19:36:47.189059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:47.188842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerDied","Data":"d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"}
Sep 16 19:36:48.195766 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:48.195738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/0.log"
Sep 16 19:36:48.196161 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:48.195830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerStarted","Data":"4bff2c6335ce382d984ff6999f60032478399f48a07975581df3cf8efb03bfbb"}
Sep 16 19:36:50.206755 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.206724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/1.log"
Sep 16 19:36:50.207209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.207063    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/0.log"
Sep 16 19:36:50.207209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.207095    2581 generic.go:356] "Generic (PLEG): container finished" podID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerID="4bff2c6335ce382d984ff6999f60032478399f48a07975581df3cf8efb03bfbb" exitCode=1
Sep 16 19:36:50.207209 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.207143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerDied","Data":"4bff2c6335ce382d984ff6999f60032478399f48a07975581df3cf8efb03bfbb"}
Sep 16 19:36:50.207323 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.207243    2581 scope.go:117] "RemoveContainer" containerID="d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"
Sep 16 19:36:50.207617 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.207597    2581 scope.go:117] "RemoveContainer" containerID="d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"
Sep 16 19:36:50.217918 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:50.217887    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-404a29-predictor-544c4b6766-nlpb8_kserve-ci-e2e-test(480acdec-db55-405b-88c6-ab5bab2a3c6f)\"" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f"
Sep 16 19:36:50.218519 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:50.218494    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_kserve-ci-e2e-test_480acdec-db55-405b-88c6-ab5bab2a3c6f_0 in pod sandbox 9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7 from index: no such id: 'd24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5'" containerID="d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"
Sep 16 19:36:50.218594 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.218528    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_kserve-ci-e2e-test_480acdec-db55-405b-88c6-ab5bab2a3c6f_0 in pod sandbox 9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7 from index: no such id: 'd24c30b902acc1a567bfba3e68f680c04e388ab1ebb6e21970d64ef539142cf5'"
Sep 16 19:36:50.410311 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:50.410282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:36:51.213443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:51.213417    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/1.log"
Sep 16 19:36:53.705816 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:53.705786    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:36:53.706242 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:53.706010    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container" containerID="cri-o://38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2" gracePeriod=30
Sep 16 19:36:53.706242 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:53.706063    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kube-rbac-proxy" containerID="cri-o://47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f" gracePeriod=30
Sep 16 19:36:54.230678 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:54.230646    2581 generic.go:356] "Generic (PLEG): container finished" podID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerID="47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f" exitCode=2
Sep 16 19:36:54.230850 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:54.230721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerDied","Data":"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"}
Sep 16 19:36:54.829121 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:54.829078    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 16 19:36:57.157569 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.157541    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:36:57.245970 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.245883    2581 generic.go:356] "Generic (PLEG): container finished" podID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerID="38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2" exitCode=0
Sep 16 19:36:57.245970 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.245942    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"
Sep 16 19:36:57.246144 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.245966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerDied","Data":"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"}
Sep 16 19:36:57.246144 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.246074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx" event={"ID":"0099380c-9fc0-4b3a-a992-60fdf238b39f","Type":"ContainerDied","Data":"648660cbd05276dddf42ebbc2ea048ff2071cdd7cd608f04ba309180070680d2"}
Sep 16 19:36:57.246144 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.246107    2581 scope.go:117] "RemoveContainer" containerID="47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"
Sep 16 19:36:57.256777 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.256752    2581 scope.go:117] "RemoveContainer" containerID="38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"
Sep 16 19:36:57.265136 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.265119    2581 scope.go:117] "RemoveContainer" containerID="3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"
Sep 16 19:36:57.272727 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.272711    2581 scope.go:117] "RemoveContainer" containerID="47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"
Sep 16 19:36:57.272975 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:57.272956    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f\": container with ID starting with 47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f not found: ID does not exist" containerID="47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"
Sep 16 19:36:57.273017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.272990    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f"} err="failed to get container status \"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f\": rpc error: code = NotFound desc = could not find container \"47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f\": container with ID starting with 47890c9e201a8e27534d89e20b89ef1806055fa577460a0e6f575252207eee4f not found: ID does not exist"
Sep 16 19:36:57.273017 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.273009    2581 scope.go:117] "RemoveContainer" containerID="38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"
Sep 16 19:36:57.273288 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:57.273270    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2\": container with ID starting with 38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2 not found: ID does not exist" containerID="38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"
Sep 16 19:36:57.273334 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.273294    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2"} err="failed to get container status \"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2\": rpc error: code = NotFound desc = could not find container \"38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2\": container with ID starting with 38826ccd6fbc66e1bf77ae6cefee3834a0a7a448b6c3d0d0b117db89017724e2 not found: ID does not exist"
Sep 16 19:36:57.273334 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.273309    2581 scope.go:117] "RemoveContainer" containerID="3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"
Sep 16 19:36:57.273523 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:36:57.273508    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c\": container with ID starting with 3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c not found: ID does not exist" containerID="3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"
Sep 16 19:36:57.273566 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.273526    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c"} err="failed to get container status \"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c\": rpc error: code = NotFound desc = could not find container \"3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c\": container with ID starting with 3bb9fb672e2793023118a954b6d1efcd88e5d33af6821846108c242f246cba3c not found: ID does not exist"
Sep 16 19:36:57.306863 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.306837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config\") pod \"0099380c-9fc0-4b3a-a992-60fdf238b39f\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") "
Sep 16 19:36:57.306973 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.306873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location\") pod \"0099380c-9fc0-4b3a-a992-60fdf238b39f\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") "
Sep 16 19:36:57.306973 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.306907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls\") pod \"0099380c-9fc0-4b3a-a992-60fdf238b39f\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") "
Sep 16 19:36:57.306973 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.306953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert\") pod \"0099380c-9fc0-4b3a-a992-60fdf238b39f\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") "
Sep 16 19:36:57.307225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.306973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwh66\" (UniqueName: \"kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66\") pod \"0099380c-9fc0-4b3a-a992-60fdf238b39f\" (UID: \"0099380c-9fc0-4b3a-a992-60fdf238b39f\") "
Sep 16 19:36:57.307225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.307186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0099380c-9fc0-4b3a-a992-60fdf238b39f" (UID: "0099380c-9fc0-4b3a-a992-60fdf238b39f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:57.307347 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.307307    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0099380c-9fc0-4b3a-a992-60fdf238b39f" (UID: "0099380c-9fc0-4b3a-a992-60fdf238b39f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:57.307394 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.307359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0099380c-9fc0-4b3a-a992-60fdf238b39f" (UID: "0099380c-9fc0-4b3a-a992-60fdf238b39f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:57.309049 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.309023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0099380c-9fc0-4b3a-a992-60fdf238b39f" (UID: "0099380c-9fc0-4b3a-a992-60fdf238b39f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:57.309138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.309072    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66" (OuterVolumeSpecName: "kube-api-access-gwh66") pod "0099380c-9fc0-4b3a-a992-60fdf238b39f" (UID: "0099380c-9fc0-4b3a-a992-60fdf238b39f"). InnerVolumeSpecName "kube-api-access-gwh66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:57.408192 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.408136    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:57.408380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.408213    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0099380c-9fc0-4b3a-a992-60fdf238b39f-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:57.408380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.408229    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0099380c-9fc0-4b3a-a992-60fdf238b39f-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:57.408380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.408241    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0099380c-9fc0-4b3a-a992-60fdf238b39f-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:57.408380 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.408252    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwh66\" (UniqueName: \"kubernetes.io/projected/0099380c-9fc0-4b3a-a992-60fdf238b39f-kube-api-access-gwh66\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:57.561514 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.561436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:36:57.566791 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:57.566764    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx"]
Sep 16 19:36:59.014498 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.014462    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:59.051584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.051544    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:36:59.051869 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.051842    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container" containerID="cri-o://05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b" gracePeriod=30
Sep 16 19:36:59.051940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.051880    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kube-rbac-proxy" containerID="cri-o://65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e" gracePeriod=30
Sep 16 19:36:59.174389 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.174369    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/1.log"
Sep 16 19:36:59.176489 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.176473    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:59.257385 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.257354    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerID="65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e" exitCode=2
Sep 16 19:36:59.257577 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.257428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerDied","Data":"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"}
Sep 16 19:36:59.258782 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.258763    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-404a29-predictor-544c4b6766-nlpb8_480acdec-db55-405b-88c6-ab5bab2a3c6f/storage-initializer/1.log"
Sep 16 19:36:59.258885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.258813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8" event={"ID":"480acdec-db55-405b-88c6-ab5bab2a3c6f","Type":"ContainerDied","Data":"9276acb59e64338c0ffe2f04edb7da9ac82c4984b653222f616ad9a99765b8b7"}
Sep 16 19:36:59.258885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.258844    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"
Sep 16 19:36:59.258885 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.258857    2581 scope.go:117] "RemoveContainer" containerID="4bff2c6335ce382d984ff6999f60032478399f48a07975581df3cf8efb03bfbb"
Sep 16 19:36:59.323503 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") pod \"480acdec-db55-405b-88c6-ab5bab2a3c6f\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") "
Sep 16 19:36:59.323628 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxsw2\" (UniqueName: \"kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2\") pod \"480acdec-db55-405b-88c6-ab5bab2a3c6f\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") "
Sep 16 19:36:59.323628 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location\") pod \"480acdec-db55-405b-88c6-ab5bab2a3c6f\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") "
Sep 16 19:36:59.323628 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert\") pod \"480acdec-db55-405b-88c6-ab5bab2a3c6f\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") "
Sep 16 19:36:59.323736 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config\") pod \"480acdec-db55-405b-88c6-ab5bab2a3c6f\" (UID: \"480acdec-db55-405b-88c6-ab5bab2a3c6f\") "
Sep 16 19:36:59.323886 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "480acdec-db55-405b-88c6-ab5bab2a3c6f" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:59.323987 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.323968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "480acdec-db55-405b-88c6-ab5bab2a3c6f" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:59.324045 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.324026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "480acdec-db55-405b-88c6-ab5bab2a3c6f" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:59.325598 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.325572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "480acdec-db55-405b-88c6-ab5bab2a3c6f" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:59.325654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.325635    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2" (OuterVolumeSpecName: "kube-api-access-lxsw2") pod "480acdec-db55-405b-88c6-ab5bab2a3c6f" (UID: "480acdec-db55-405b-88c6-ab5bab2a3c6f"). InnerVolumeSpecName "kube-api-access-lxsw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:59.414606 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.414577    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" path="/var/lib/kubelet/pods/0099380c-9fc0-4b3a-a992-60fdf238b39f/volumes"
Sep 16 19:36:59.424940 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.424921    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:59.425039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.424942    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:59.425039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.424952    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480acdec-db55-405b-88c6-ab5bab2a3c6f-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:59.425039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.424961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxsw2\" (UniqueName: \"kubernetes.io/projected/480acdec-db55-405b-88c6-ab5bab2a3c6f-kube-api-access-lxsw2\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:59.425039 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.424969    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480acdec-db55-405b-88c6-ab5bab2a3c6f-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:36:59.588481 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.588397    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:59.591554 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.591522    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8"]
Sep 16 19:36:59.750191 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:36:59.750135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:37:01.413751 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:01.413718    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" path="/var/lib/kubelet/pods/480acdec-db55-405b-88c6-ab5bab2a3c6f/volumes"
Sep 16 19:37:03.699988 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.699963    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:37:03.863339 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crhzp\" (UniqueName: \"kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp\") pod \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") "
Sep 16 19:37:03.863519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert\") pod \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") "
Sep 16 19:37:03.863519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location\") pod \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") "
Sep 16 19:37:03.863519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls\") pod \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") "
Sep 16 19:37:03.863519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config\") pod \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\" (UID: \"5e3ee073-7d23-49f0-840c-4d97cdf9731b\") "
Sep 16 19:37:03.863856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e3ee073-7d23-49f0-840c-4d97cdf9731b" (UID: "5e3ee073-7d23-49f0-840c-4d97cdf9731b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:03.863856 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863842    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e3ee073-7d23-49f0-840c-4d97cdf9731b" (UID: "5e3ee073-7d23-49f0-840c-4d97cdf9731b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:03.864034 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.863881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e3ee073-7d23-49f0-840c-4d97cdf9731b" (UID: "5e3ee073-7d23-49f0-840c-4d97cdf9731b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:03.865477 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.865451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp" (OuterVolumeSpecName: "kube-api-access-crhzp") pod "5e3ee073-7d23-49f0-840c-4d97cdf9731b" (UID: "5e3ee073-7d23-49f0-840c-4d97cdf9731b"). InnerVolumeSpecName "kube-api-access-crhzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:03.865610 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.865588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e3ee073-7d23-49f0-840c-4d97cdf9731b" (UID: "5e3ee073-7d23-49f0-840c-4d97cdf9731b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:03.964421 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.964390    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhzp\" (UniqueName: \"kubernetes.io/projected/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-api-access-crhzp\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:37:03.964421 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.964419    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-cabundle-cert\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:37:03.964609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.964430    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kserve-provision-location\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:37:03.964609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.964440    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e3ee073-7d23-49f0-840c-4d97cdf9731b-proxy-tls\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:37:03.964609 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:03.964449    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e3ee073-7d23-49f0-840c-4d97cdf9731b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-142.ec2.internal\" DevicePath \"\""
Sep 16 19:37:04.283654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.283558    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerID="05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b" exitCode=0
Sep 16 19:37:04.283654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.283613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"
Sep 16 19:37:04.283654 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.283632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerDied","Data":"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"}
Sep 16 19:37:04.283910 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.283690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt" event={"ID":"5e3ee073-7d23-49f0-840c-4d97cdf9731b","Type":"ContainerDied","Data":"1a86ee7c04d5a95415c4cb20433115251002e2ff9c3a23481d052a50a1f43619"}
Sep 16 19:37:04.283910 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.283717    2581 scope.go:117] "RemoveContainer" containerID="65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"
Sep 16 19:37:04.293397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.293379    2581 scope.go:117] "RemoveContainer" containerID="05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"
Sep 16 19:37:04.301059 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.301036    2581 scope.go:117] "RemoveContainer" containerID="8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"
Sep 16 19:37:04.305618 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.305597    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:37:04.309248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.309225    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt"]
Sep 16 19:37:04.309455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.309438    2581 scope.go:117] "RemoveContainer" containerID="65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"
Sep 16 19:37:04.309721 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:37:04.309702    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e\": container with ID starting with 65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e not found: ID does not exist" containerID="65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"
Sep 16 19:37:04.309768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.309730    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e"} err="failed to get container status \"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e\": rpc error: code = NotFound desc = could not find container \"65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e\": container with ID starting with 65e7a2e27a0ae3e24225fdf15ea13123b9e6f00d2e95b8d87a4852766b53f31e not found: ID does not exist"
Sep 16 19:37:04.309768 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.309749    2581 scope.go:117] "RemoveContainer" containerID="05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"
Sep 16 19:37:04.309991 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:37:04.309973    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b\": container with ID starting with 05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b not found: ID does not exist" containerID="05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"
Sep 16 19:37:04.310032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.309997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b"} err="failed to get container status \"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b\": rpc error: code = NotFound desc = could not find container \"05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b\": container with ID starting with 05bd65a763e079f10a8a8ca36d06c79bb79a5b01d0fc3cb4a271c5eafd0e765b not found: ID does not exist"
Sep 16 19:37:04.310032 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.310012    2581 scope.go:117] "RemoveContainer" containerID="8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"
Sep 16 19:37:04.310244 ip-10-0-130-142 kubenswrapper[2581]: E0916 19:37:04.310227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21\": container with ID starting with 8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21 not found: ID does not exist" containerID="8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"
Sep 16 19:37:04.310292 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:04.310248    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21"} err="failed to get container status \"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21\": rpc error: code = NotFound desc = could not find container \"8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21\": container with ID starting with 8e0de8a5e05bd3d0126432ba028ab6bb339f28a7f243bd4b2d3d2cce73f61e21 not found: ID does not exist"
Sep 16 19:37:05.414584 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:05.414554    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" path="/var/lib/kubelet/pods/5e3ee073-7d23-49f0-840c-4d97cdf9731b/volumes"
Sep 16 19:37:17.800170 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:17.800127    2581 scope.go:117] "RemoveContainer" containerID="268560be2170a4e9ec180bb5194d3a9b2d63320b7c7488e57f2f9b962370adaa"
Sep 16 19:37:17.808077 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:17.808058    2581 scope.go:117] "RemoveContainer" containerID="390aafeab8756e07e2dd19f07c55ce1a26fe79971d8fb7535a0a1a6faab331ea"
Sep 16 19:37:17.815402 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:17.815380    2581 scope.go:117] "RemoveContainer" containerID="1a998ccad87b9496794a74c9b155e77af348ffdb10e4c1134bd57ce5a1a990b8"
Sep 16 19:37:41.524506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524430    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rwfh7/must-gather-lqd9w"]
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524759    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524772    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524780    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524785    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524799    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kube-rbac-proxy"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524815    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kube-rbac-proxy"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524822    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kube-rbac-proxy"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524828    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kube-rbac-proxy"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524833    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524848    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524853    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524866    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524913    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524928    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099380c-9fc0-4b3a-a992-60fdf238b39f" containerName="kube-rbac-proxy"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524935    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kserve-container"
Sep 16 19:37:41.524937 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524942    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="480acdec-db55-405b-88c6-ab5bab2a3c6f" containerName="storage-initializer"
Sep 16 19:37:41.525606 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.524949    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3ee073-7d23-49f0-840c-4d97cdf9731b" containerName="kube-rbac-proxy"
Sep 16 19:37:41.527952 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.527934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.530629 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.530608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rwfh7\"/\"kube-root-ca.crt\""
Sep 16 19:37:41.540455 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.540437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rwfh7\"/\"openshift-service-ca.crt\""
Sep 16 19:37:41.540686 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.540669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rwfh7\"/\"default-dockercfg-wx2v9\""
Sep 16 19:37:41.545057 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.545026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/must-gather-lqd9w"]
Sep 16 19:37:41.671506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.671470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4593b0e0-e367-472d-8205-be40fcac496d-must-gather-output\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.671506 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.671513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmk8k\" (UniqueName: \"kubernetes.io/projected/4593b0e0-e367-472d-8205-be40fcac496d-kube-api-access-qmk8k\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.772211 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.772171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4593b0e0-e367-472d-8205-be40fcac496d-must-gather-output\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.772339 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.772226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmk8k\" (UniqueName: \"kubernetes.io/projected/4593b0e0-e367-472d-8205-be40fcac496d-kube-api-access-qmk8k\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.772466 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.772445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4593b0e0-e367-472d-8205-be40fcac496d-must-gather-output\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.780754 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.780693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmk8k\" (UniqueName: \"kubernetes.io/projected/4593b0e0-e367-472d-8205-be40fcac496d-kube-api-access-qmk8k\") pod \"must-gather-lqd9w\" (UID: \"4593b0e0-e367-472d-8205-be40fcac496d\") " pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:41.848569 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:41.848522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/must-gather-lqd9w"
Sep 16 19:37:42.119829 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:42.119802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/must-gather-lqd9w"]
Sep 16 19:37:42.122462 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:37:42.122431    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4593b0e0_e367_472d_8205_be40fcac496d.slice/crio-ca34f3c372c539a2bb0f89e086396879e13b65bda776dfa3f243414d9a904ce9 WatchSource:0}: Error finding container ca34f3c372c539a2bb0f89e086396879e13b65bda776dfa3f243414d9a904ce9: Status 404 returned error can't find the container with id ca34f3c372c539a2bb0f89e086396879e13b65bda776dfa3f243414d9a904ce9
Sep 16 19:37:42.124499 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:42.124484    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:37:42.447896 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:42.447808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/must-gather-lqd9w" event={"ID":"4593b0e0-e367-472d-8205-be40fcac496d","Type":"ContainerStarted","Data":"ca34f3c372c539a2bb0f89e086396879e13b65bda776dfa3f243414d9a904ce9"}
Sep 16 19:37:43.453883 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:43.453842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/must-gather-lqd9w" event={"ID":"4593b0e0-e367-472d-8205-be40fcac496d","Type":"ContainerStarted","Data":"9866dbf4a47164451cace20982d9a8c2d9e776ae3145ef5fb04dfc929f6e0027"}
Sep 16 19:37:43.454309 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:43.453896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/must-gather-lqd9w" event={"ID":"4593b0e0-e367-472d-8205-be40fcac496d","Type":"ContainerStarted","Data":"2bef8836fccb6016303a377d04018359b62abcc7b072ff34d964081b66366b59"}
Sep 16 19:37:43.471773 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:43.471724    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rwfh7/must-gather-lqd9w" podStartSLOduration=1.676386747 podStartE2EDuration="2.471707383s" totalImagesPullingTime="795.320636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:43.469448809 +0000 UTC m=+3238.531788060" watchObservedRunningTime="2026-09-16 19:37:43.471707383 +0000 UTC m=+3238.534046633"
Sep 16 19:37:44.592304 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:44.592272    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xqtnz_9528a95a-967e-48e3-9caf-5604a16ffa0c/global-pull-secret-syncer/0.log"
Sep 16 19:37:44.704397 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:44.704359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6nm7n_d2d54608-aad8-4ae1-a382-29808406a431/konnectivity-agent/0.log"
Sep 16 19:37:44.817357 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:44.817327    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-142.ec2.internal_58d471a5ef85e5ea0c5b87f8f80839ec/haproxy/0.log"
Sep 16 19:37:49.348860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.348831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-7wb68_3db0cae2-5bdc-4727-bf51-09cc3d97bb47/monitoring-plugin/0.log"
Sep 16 19:37:49.470916 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.470885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxcdf_08ab8584-5ac2-4ab2-9bf9-007c943a0750/node-exporter/0.log"
Sep 16 19:37:49.495859 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.495824    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxcdf_08ab8584-5ac2-4ab2-9bf9-007c943a0750/kube-rbac-proxy/0.log"
Sep 16 19:37:49.521422 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.521387    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxcdf_08ab8584-5ac2-4ab2-9bf9-007c943a0750/init-textfile/0.log"
Sep 16 19:37:49.948801 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.948751    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-km4cp_0f61d792-9cc5-405f-91e4-b4a66878e63a/prometheus-operator/0.log"
Sep 16 19:37:49.968377 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.968350    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-km4cp_0f61d792-9cc5-405f-91e4-b4a66878e63a/kube-rbac-proxy/0.log"
Sep 16 19:37:49.996694 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:49.996652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-jzt7z_b79d70dc-5358-4de3-b19a-0fab21815a56/prometheus-operator-admission-webhook/0.log"
Sep 16 19:37:50.110779 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.110750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/thanos-query/0.log"
Sep 16 19:37:50.135518 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.135479    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/kube-rbac-proxy-web/0.log"
Sep 16 19:37:50.162931 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.162897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/kube-rbac-proxy/0.log"
Sep 16 19:37:50.193248 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.193195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/prom-label-proxy/0.log"
Sep 16 19:37:50.217000 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.216881    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/kube-rbac-proxy-rules/0.log"
Sep 16 19:37:50.248833 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:50.248800    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54757d56d8-k6qw8_c5976f70-bb7b-49e2-aca5-01ad137c495a/kube-rbac-proxy-metrics/0.log"
Sep 16 19:37:52.458265 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.458221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ffddb9bf5-j5nqw_03e36c06-6084-484d-839b-7f3f276fe9d9/console/0.log"
Sep 16 19:37:52.503350 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.503298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-v4frq_de8dde70-47db-41ee-9200-2352aedf2b65/download-server/0.log"
Sep 16 19:37:52.714741 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.714657    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"]
Sep 16 19:37:52.721339 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.721307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.726128 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.726065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"]
Sep 16 19:37:52.874138 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.874095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwmj\" (UniqueName: \"kubernetes.io/projected/48758d10-a451-4c94-ad9e-87df509d8cac-kube-api-access-tqwmj\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.874440 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.874416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-proc\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.874759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.874728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-lib-modules\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.874860 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.874789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-podres\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.874910 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.874860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-sys\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.975695 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.975602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-lib-modules\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.975968 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.975945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-podres\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976088 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-podres\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976175 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.975836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-lib-modules\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976329 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-sys\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976445 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-sys\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976559 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwmj\" (UniqueName: \"kubernetes.io/projected/48758d10-a451-4c94-ad9e-87df509d8cac-kube-api-access-tqwmj\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976668 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-proc\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.976849 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.976829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48758d10-a451-4c94-ad9e-87df509d8cac-proc\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:52.984833 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:52.984805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwmj\" (UniqueName: \"kubernetes.io/projected/48758d10-a451-4c94-ad9e-87df509d8cac-kube-api-access-tqwmj\") pod \"perf-node-gather-daemonset-ddn7v\" (UID: \"48758d10-a451-4c94-ad9e-87df509d8cac\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:53.035912 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.035882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:53.332647 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.332616    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"]
Sep 16 19:37:53.336538 ip-10-0-130-142 kubenswrapper[2581]: W0916 19:37:53.336498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48758d10_a451_4c94_ad9e_87df509d8cac.slice/crio-e59711832fca2c9bee55db62777f25123e42c0744f4284e4cb0204ceb2a31ade WatchSource:0}: Error finding container e59711832fca2c9bee55db62777f25123e42c0744f4284e4cb0204ceb2a31ade: Status 404 returned error can't find the container with id e59711832fca2c9bee55db62777f25123e42c0744f4284e4cb0204ceb2a31ade
Sep 16 19:37:53.552755 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.552720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v" event={"ID":"48758d10-a451-4c94-ad9e-87df509d8cac","Type":"ContainerStarted","Data":"4fc606a451101dc94a892e1cf9ca7bfa322b7efdf703ff32f67391aa0ad6ad66"}
Sep 16 19:37:53.553137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.552774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v" event={"ID":"48758d10-a451-4c94-ad9e-87df509d8cac","Type":"ContainerStarted","Data":"e59711832fca2c9bee55db62777f25123e42c0744f4284e4cb0204ceb2a31ade"}
Sep 16 19:37:53.553137 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.552804    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:53.568519 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:53.568468    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v" podStartSLOduration=1.568451737 podStartE2EDuration="1.568451737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:53.567870025 +0000 UTC m=+3248.630209277" watchObservedRunningTime="2026-09-16 19:37:53.568451737 +0000 UTC m=+3248.630790988"
Sep 16 19:37:54.027342 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:54.027311    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pt88_519727fa-2299-44e2-a7bd-65a55a78824d/dns/0.log"
Sep 16 19:37:54.049638 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:54.049609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pt88_519727fa-2299-44e2-a7bd-65a55a78824d/kube-rbac-proxy/0.log"
Sep 16 19:37:54.176702 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:54.176673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfnjj_990619d1-4ef1-4d44-943f-b7236ff5c472/dns-node-resolver/0.log"
Sep 16 19:37:54.739672 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:54.739636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f9bd6d57-bggql_f62a0da4-c4e6-4c89-a9d7-6a3654803849/registry/0.log"
Sep 16 19:37:54.801659 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:54.801628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bpd6c_f4f64602-4dbb-484f-ba94-3c722eb72730/node-ca/0.log"
Sep 16 19:37:56.035737 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:56.035709    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tdz6v_893b03c4-abf1-42e1-ba25-1c1e5e1be64a/serve-healthcheck-canary/0.log"
Sep 16 19:37:56.622393 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:56.622365    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s55fk_b34d79c2-5815-43a6-aea1-8f328e81c379/kube-rbac-proxy/0.log"
Sep 16 19:37:56.643323 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:56.643287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s55fk_b34d79c2-5815-43a6-aea1-8f328e81c379/exporter/0.log"
Sep 16 19:37:56.665767 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:56.665743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s55fk_b34d79c2-5815-43a6-aea1-8f328e81c379/extractor/0.log"
Sep 16 19:37:59.101268 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.101232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-72gdg_e4fe785e-03d3-4530-af47-f8bdd6d8ab68/server/0.log"
Sep 16 19:37:59.571285 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.571199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-ddn7v"
Sep 16 19:37:59.648220 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.648183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-w8wjj_094141b9-07c2-4d85-af6f-d49fc847b434/manager/0.log"
Sep 16 19:37:59.883416 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.883385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-l9fj9_859c5c3f-a95d-4dcf-91ad-ab500a24f07f/seaweedfs/0.log"
Sep 16 19:37:59.908929 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.908897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-dx7gg_caaec21e-ce8e-4e38-bb1b-2c0fd8a55ee2/seaweedfs-tls-custom/0.log"
Sep 16 19:37:59.934978 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:37:59.934950    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-t5xht_ee67e50f-0359-4802-be36-9316f137c7a7/seaweedfs-tls-serving/0.log"
Sep 16 19:38:04.380759 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:04.380723    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-p7lhc_b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79/migrator/0.log"
Sep 16 19:38:04.402795 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:04.402761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-p7lhc_b1a6102f-c3e9-48ae-9a17-9dd17a5f3c79/graceful-termination/0.log"
Sep 16 19:38:04.770802 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:04.770720    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lhw7j_cc51e700-c58d-48ce-b099-e448440ebf32/kube-storage-version-migrator-operator/1.log"
Sep 16 19:38:04.772901 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:04.772872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lhw7j_cc51e700-c58d-48ce-b099-e448440ebf32/kube-storage-version-migrator-operator/0.log"
Sep 16 19:38:06.145286 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.145252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:38:06.170750 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.170722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/egress-router-binary-copy/0.log"
Sep 16 19:38:06.195256 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.195225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/cni-plugins/0.log"
Sep 16 19:38:06.227175 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.227134    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/bond-cni-plugin/0.log"
Sep 16 19:38:06.250991 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.250959    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/routeoverride-cni/0.log"
Sep 16 19:38:06.273868 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.273842    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/whereabouts-cni-bincopy/0.log"
Sep 16 19:38:06.295710 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.295680    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tv858_b581ff8c-1c56-4015-89bf-b67d85e91e6b/whereabouts-cni/0.log"
Sep 16 19:38:06.378443 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.378393    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n87v8_24eaddca-b749-49b7-9641-3ef3070ad410/kube-multus/0.log"
Sep 16 19:38:06.509571 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.509495    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9drp_e1632371-d9cd-4766-a6c0-521a31af4fe6/network-metrics-daemon/0.log"
Sep 16 19:38:06.528823 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:06.528786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9drp_e1632371-d9cd-4766-a6c0-521a31af4fe6/kube-rbac-proxy/0.log"
Sep 16 19:38:07.286965 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.286918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/ovn-controller/0.log"
Sep 16 19:38:07.306545 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.306514    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/ovn-acl-logging/0.log"
Sep 16 19:38:07.330902 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.330874    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/kube-rbac-proxy-node/0.log"
Sep 16 19:38:07.355480 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.355452    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:38:07.380468 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.380447    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/northd/0.log"
Sep 16 19:38:07.404404 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.404375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/nbdb/0.log"
Sep 16 19:38:07.426812 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.426784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/sbdb/0.log"
Sep 16 19:38:07.495225 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:07.495190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knw5n_65f3ab43-1d3e-4c70-a727-2ce2f5a67f0c/ovnkube-controller/0.log"
Sep 16 19:38:08.956858 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:08.956824    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-2hwqk_24bfc5ee-4b9b-460b-95c8-7ea02e5b56f2/check-endpoints/0.log"
Sep 16 19:38:09.009018 ip-10-0-130-142 kubenswrapper[2581]: I0916 19:38:09.008987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h7gn6_46c18fd8-6738-4435-a7dc-6180ef83b6dd/network-check-target-container/0.log"