Jul 28 12:34:51.962271 ip-10-0-138-35 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:34:52.377865 ip-10-0-138-35 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:52.377865 ip-10-0-138-35 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:34:52.377865 ip-10-0-138-35 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:52.377865 ip-10-0-138-35 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:34:52.377865 ip-10-0-138-35 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:52.378682 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.377949    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384333    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384356    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384360    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384363    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384366    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.384359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384369    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384372    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384375    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384377    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384380    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384382    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384385    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384387    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384390    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384393    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384396    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384398    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384401    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384403    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384408    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384412    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384415    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384418    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384420    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384424    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.384597 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384427    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384430    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384432    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384435    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384438    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384440    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384442    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384449    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384451    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384454    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384456    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384458    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384461    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384464    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384467    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384469    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384472    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384474    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384476    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384479    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.385069 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384481    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384483    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384486    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384488    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384491    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384493    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384495    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384498    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384500    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384502    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384505    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384507    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384510    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384513    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384516    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384519    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384522    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384526    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384528    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.385535 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384531    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384534    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384536    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384539    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384541    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384544    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384548    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384550    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384553    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384555    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384557    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384560    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384562    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384565    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384567    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384569    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384572    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384574    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384576    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384579    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.385998 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384581    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384584    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384586    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384589    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384591    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384594    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384597    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384599    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384601    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384604    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384606    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384609    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384611    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384615    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.384619    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385330    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385337    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385342    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385345    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.386461 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385347    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385350    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385353    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385355    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385357    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385360    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385362    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385365    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385367    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385370    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385372    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385375    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385377    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385380    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385384    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385387    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385390    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385393    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385395    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.386904 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385398    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385400    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385403    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385406    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385408    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385411    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385413    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385416    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385418    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385420    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385422    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385426    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385429    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385431    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385434    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385436    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385438    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385441    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385443    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385446    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385448    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.387377 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385451    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385453    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385455    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385459    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385462    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385465    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385468    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385471    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385474    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385477    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385479    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385482    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385485    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385487    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385490    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385492    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385495    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385497    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385500    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385502    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.387869 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385505    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385508    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385511    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385515    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385518    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385520    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385523    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385525    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385527    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385530    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385532    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385535    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385537    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385540    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385542    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385545    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385548    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385552    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385554    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.388362 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385560    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385562    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385565    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385567    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385569    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385572    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385574    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385579    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385583    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385585    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385588    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385590    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385592    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385595    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385598    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.385601    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386002    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386008    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386011    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386014    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.388805 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386016    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386019    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386022    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386024    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386027    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386030    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386032    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386035    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386037    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386040    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386042    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386045    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386048    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386050    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386052    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386055    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386057    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386060    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386062    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386065    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.389305 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386067    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386071    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386075    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386078    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386080    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386083    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386085    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386088    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386091    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386093    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386096    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386098    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386101    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386104    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386106    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386108    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386111    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386113    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386115    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.389898 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386118    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386121    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386124    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386127    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386129    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386131    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386134    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386136    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386139    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386141    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386144    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386146    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386149    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386152    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386155    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386158    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386160    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386162    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386165    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386167    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.390355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386170    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386172    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386174    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386178    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386180    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386183    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386185    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386187    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386190    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386192    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386195    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386198    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386200    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386202    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386204    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386207    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386209    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386211    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386214    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.390809 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386217    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386220    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386223    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386226    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386228    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386231    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386234    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386237    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386239    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386242    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386244    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386247    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386249    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386251    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386254    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386256    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386259    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386335    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386351    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386359    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:34:52.391310 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386365    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386371    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386375    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386382    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386386    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386389    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386392    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386395    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386398    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386401    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386404    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386406    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386409    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386413    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386417    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386421    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386424    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386427    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386430    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386433    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386436    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386439    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386442    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:34:52.391771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386445    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386448    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386450    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386453    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386457    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386459    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386463    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386466    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386469    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386472    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386474    2558 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386477    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386481    2558 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386483    2558 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386486    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386489    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386491    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386495    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386497    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386500    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386503    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386505    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386508    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386511    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386516    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:34:52.392358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386519    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386521    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386524    2558 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386528    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386531    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386534    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386537    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386539    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386542    2558 flags.go:64] FLAG: --help="false"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386545    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386547    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386550    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386553    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386556    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386559    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386561    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386564    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386567    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386569    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386572    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386575    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386577    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386580    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386583    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:34:52.392957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386585    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386588    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386590    2558 flags.go:64] FLAG: --lock-file=""
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386593    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386596    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386598    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386603    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386608    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386610    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386613    2558 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386616    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386619    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386622    2558 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386625    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386629    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386632    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386636    2558 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386638    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386641    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386644    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386647    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386649    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386652    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386655    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386662    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:34:52.393537 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386665    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386668    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386671    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386673    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386679    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386682    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386684    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386687    2558 flags.go:64] FLAG: --port="10250"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386690    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386693    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04bebc6e5c8103fcb"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386695    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386698    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386700    2558 flags.go:64] FLAG: --register-node="true"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386703    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386707    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386711    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386716    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386718    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386722    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386725    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386728    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386730    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386733    2558 flags.go:64] FLAG: --runonce="false"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386735    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386738    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:34:52.394213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386741    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386743    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386746    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386749    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386752    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386755    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386758    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386761    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386764    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386766    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386770    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386772    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386775    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386779    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386782    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386785    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386789    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386791    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386794    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386796    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386799    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386802    2558 flags.go:64] FLAG: --v="2"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386807    2558 flags.go:64] FLAG: --version="false"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386813    2558 flags.go:64] FLAG: --vmodule=""
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386817    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:34:52.394796 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.386820    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386938    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386942    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386945    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386948    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386951    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386954    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386956    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386959    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386961    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386964    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386966    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386984    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386987    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386989    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386991    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386994    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386996    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.386999    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387001    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.395388 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387003    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387006    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387008    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387011    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387013    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387015    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387018    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387020    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387023    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387026    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387031    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387034    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387036    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387040    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387043    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387046    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387049    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387052    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387054    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.395890 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387057    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387059    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387061    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387064    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387066    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387068    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387071    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387073    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387076    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387078    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387081    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387083    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387086    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387088    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387090    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387092    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387095    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387098    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387100    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387102    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.396355 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387105    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387107    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387111    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387115    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387117    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387120    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387123    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387125    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387127    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387130    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387132    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387136    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387139    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387142    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387145    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387147    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387150    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387152    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387154    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387157    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.396830 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387159    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387161    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387164    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387166    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387169    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387171    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387173    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387176    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387178    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387180    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387184    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387187    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387189    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387192    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387195    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387200    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387202    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387205    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387208    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387210    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.397301 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.387213    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.387787    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.394633    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.394651    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.394670    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.394678    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394736    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394741    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394744    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394747    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394750    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394753    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394755    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394758    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394761    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.397768 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394763    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394766    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394769    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394772    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394774    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394777    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394779    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394781    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394784    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394786    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394789    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394791    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394794    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394796    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394799    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394802    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394805    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394807    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394809    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.398155 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394812    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394817    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394821    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394824    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394827    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394830    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394833    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394835    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394838    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394840    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394843    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394845    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394848    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394850    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394852    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394855    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394857    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394860    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394862    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.398591 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394864    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394866    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394869    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394871    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394875    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394878    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394881    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394884    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394887    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394889    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394892    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394895    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394897    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394900    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394903    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394905    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394908    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394910    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394912    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394915    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.399051 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394917    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394919    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394922    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394924    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394926    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394928    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394931    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394933    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394936    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394938    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394940    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394943    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394945    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394947    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394950    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394952    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394954    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394957    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394960    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394963    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.399522 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394967    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394987    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394991    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394996    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.394998    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395001    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395004    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395006    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395009    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395011    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395014    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395016    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.395021    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395135    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395139    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:52.400054 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395142    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395145    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395148    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395150    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395153    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395156    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395158    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395160    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395163    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395165    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395167    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395170    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395173    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395175    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395178    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395180    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395183    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395186    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395188    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395190    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:52.400412 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395193    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395196    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395199    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395202    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395204    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395207    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395209    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395211    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395214    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395216    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395218    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395221    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395223    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395226    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395228    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395230    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395234    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395237    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395240    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:52.400877 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395242    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395245    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395247    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395250    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395252    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395255    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395257    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395259    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395263    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395266    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395268    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395270    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395273    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395275    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395277    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395282    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395285    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395288    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395291    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:52.401329 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395293    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395296    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395298    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395301    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395303    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395305    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395308    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395310    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395312    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395315    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395317    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395319    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395322    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395324    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395326    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395328    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395330    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395333    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395335    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395337    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:52.401785 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395339    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395342    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395345    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395347    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395349    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395352    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395354    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395357    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395359    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395362    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395364    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395367    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395369    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395372    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395374    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395377    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395379    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395381    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:52.402342 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:52.395383    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.395388    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.395492    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.398226    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.399573    2558 server.go:1019] "Starting client certificate rotation"
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.399668    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.399705    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:34:52.402765 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.401802    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:34:52.407357 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.407333    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:34:52.417739 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.417716    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:34:52.419651 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.419637    2558 log.go:25] "Validated CRI v1 image API"
Jul 28 12:34:52.420425 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.420407    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:34:52.422738 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.422721    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86476b6d-f3ed-4d22-b6b3-4a83cad25b31:/dev/nvme0n1p3 8ce49ce8-435b-405b-84b3-d68cacf42241:/dev/nvme0n1p4]
Jul 28 12:34:52.422783 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.422739    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:34:52.427632 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.427519    2558 manager.go:216] Machine: {Timestamp:2026-07-28 12:34:52.426254872 +0000 UTC m=+0.339626502 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3240184 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276dd9c4914793f227fa8987a48025 SystemUUID:ec276dd9-c491-4793-f227-fa8987a48025 BootID:8222a3b6-065b-42ac-aca4-1ac7309494d0 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:9d:f3:0b:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:9d:f3:0b:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:e6:50:80:45:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:34:52.427632 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.427621    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:34:52.427751 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.427731    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:34:52.429321 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.429307    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:34:52.430843 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.430817    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:34:52.430983 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.430847    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:34:52.431506 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.431496    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:34:52.431535 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.431510    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:34:52.431535 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.431517    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:34:52.432108 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.432098    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:34:52.433740 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.433730    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:34:52.435623 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435611    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:34:52.435736 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435727    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:34:52.435768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435755    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:34:52.435768 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435766    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:34:52.435822 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435780    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:34:52.435822 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435788    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:34:52.435822 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.435799    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:34:52.436233 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.436220    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:34:52.436233 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.436242    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:34:52.438363 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.438348    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:34:52.440152 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.440134    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:34:52.440214 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.440168    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:34:52.442149 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442136    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442153    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442158    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442164    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442173    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442178    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442186    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442192    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442198    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442203    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442212    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:34:52.442237 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442218    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:34:52.442952 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442942    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:34:52.442952 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.442951    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:34:52.446902 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.446885    2558 server.go:1268] "Started kubelet"
Jul 28 12:34:52.447117 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.447041    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:34:52.447214 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.447137    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:34:52.447214 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.447142    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:34:52.447297 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.447258    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:34:52.447404 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.447383    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:34:52.447541 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.447520    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:34:52.447771 ip-10-0-138-35 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:34:52.448960 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.448943    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:34:52.449204 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.449191    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:34:52.453065 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.453047    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:34:52.453867 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.453849    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jzkms"
Jul 28 12:34:52.454439 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.453589    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c67439765eaa68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-07-28 12:34:52.446853736 +0000 UTC m=+0.360225365,LastTimestamp:2026-07-28 12:34:52.446853736 +0000 UTC m=+0.360225365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}"
Jul 28 12:34:52.456381 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.456366    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:34:52.456442 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.456380    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:34:52.456442 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.456434    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.456954 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.456937    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:34:52.456954 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.456937    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:34:52.457103 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.456965    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:34:52.457103 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457067    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:34:52.457103 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457078    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:34:52.457227 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457124    2558 factory.go:55] Registering systemd factory
Jul 28 12:34:52.457227 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457139    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:34:52.457376 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457358    2558 factory.go:153] Registering CRI-O factory
Jul 28 12:34:52.457376 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457376    2558 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:34:52.457516 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457428    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:34:52.457516 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457450    2558 factory.go:103] Registering Raw factory
Jul 28 12:34:52.457516 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457464    2558 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:34:52.457803 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.457792    2558 manager.go:349] Starting recovery of all containers
Jul 28 12:34:52.459535 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.459505    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:34:52.459623 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.459553    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:34:52.460098 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.460076    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jzkms"
Jul 28 12:34:52.468542 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.468395    2558 manager.go:354] Recovery completed
Jul 28 12:34:52.474790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.474775    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.478595 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.478576    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.478657 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.478613    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.478657 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.478625    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.479162 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.479147    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:34:52.479162 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.479160    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:34:52.479284 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.479178    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:34:52.479284 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.479235    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.480148 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.480134    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.480189 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.480165    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.480189 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.480176    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.481276 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.481262    2558 policy_none.go:49] "None policy: Start"
Jul 28 12:34:52.481333 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.481281    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:34:52.481333 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.481294    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:34:52.482398 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.482386    2558 policy_none.go:47] "Start"
Jul 28 12:34:52.483255 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.483192    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c674397843095d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-07-28 12:34:52.478597469 +0000 UTC m=+0.391969102,LastTimestamp:2026-07-28 12:34:52.478597469 +0000 UTC m=+0.391969102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}"
Jul 28 12:34:52.517712 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.517551    2558 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.517732    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.517745    2558 server.go:87] "Starting device plugin registration server"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.517947    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.517955    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.518068    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.518147    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.518156    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.518690    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:34:52.525578 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.518729    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.582354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.582308    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:34:52.583584 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.583568    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:34:52.583706 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.583613    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:34:52.583706 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.583636    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:34:52.583706 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.583667    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:34:52.585596 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.585575    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:52.618681 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.618660    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.619513 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.619496    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.619607 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.619532    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.619607 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.619546    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.619607 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.619574    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.627188 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.627172    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.627254 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.627195    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-35.ec2.internal\": node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.643804 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.643753    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.684352 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.684326    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"]
Jul 28 12:34:52.684451 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.684391    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.685908 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.685894    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.685990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.685922    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.685990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.685931    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.687105 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687094    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.687245 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687232    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.687280 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687262    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.687790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687775    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.687790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687786    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.687903 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687804    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.687903 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687808    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.687903 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687820    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.687903 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.687823    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.688918 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.688904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.688986 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.688929    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:52.689509 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.689496    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:52.689572 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.689518    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:52.689572 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.689527    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:52.714537 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.714520    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.ec2.internal\" not found" node="ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.718533 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.718518    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.ec2.internal\" not found" node="ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.744528 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.744509    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.758886 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.758862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.758964 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.758891    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12610501867f6e946c0cb2390ee3cc4f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"12610501867f6e946c0cb2390ee3cc4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.758964 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.758906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.844870 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.844827    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:52.859159 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.859269 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.859269 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12610501867f6e946c0cb2390ee3cc4f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"12610501867f6e946c0cb2390ee3cc4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.859269 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859232    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.859269 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4306b1e970225de593bcccc578a5aca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"c4306b1e970225de593bcccc578a5aca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.859269 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:52.859233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12610501867f6e946c0cb2390ee3cc4f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"12610501867f6e946c0cb2390ee3cc4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:52.945640 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:52.945572    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.017135 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.017113    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:53.022048 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.022032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:53.046353 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:53.046319    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.146884 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:53.146841    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.247406 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:53.247344    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.347954 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:53.347917    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.399406 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.399353    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:34:53.399849 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.399527    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:34:53.426932 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.426902    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:53.448592 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:53.448567    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found"
Jul 28 12:34:53.466272 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.466242    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:29:53 +0000 UTC" deadline="2028-02-07 19:46:54.143149275 +0000 UTC"
Jul 28 12:34:53.466272 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.466268    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13423h12m0.676884361s"
Jul 28 12:34:53.511314 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.511244    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:53.557315 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.557288    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:53.568953 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.568933    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:34:53.569929 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.569916    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"
Jul 28 12:34:53.577256 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:53.577231    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4306b1e970225de593bcccc578a5aca.slice/crio-576c346be769876fe3d4a4719820f97b24ce87fc0411491204f285f333730436 WatchSource:0}: Error finding container 576c346be769876fe3d4a4719820f97b24ce87fc0411491204f285f333730436: Status 404 returned error can't find the container with id 576c346be769876fe3d4a4719820f97b24ce87fc0411491204f285f333730436
Jul 28 12:34:53.578354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.578337    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:34:53.580730 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.580712    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:34:53.588054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.588017    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"c4306b1e970225de593bcccc578a5aca","Type":"ContainerStarted","Data":"576c346be769876fe3d4a4719820f97b24ce87fc0411491204f285f333730436"}
Jul 28 12:34:53.816315 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:53.816221    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:54.346398 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.346372    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:54.437174 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.437139    2558 apiserver.go:52] "Watching apiserver"
Jul 28 12:34:54.446172 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.446147    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:34:54.447396 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.447366    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vfpkz","openshift-dns/node-resolver-sq4xr","openshift-image-registry/node-ca-t724j","openshift-multus/multus-additional-cni-plugins-k6z5b","openshift-multus/multus-fbc2s","openshift-network-operator/iptables-alerter-fw29v","kube-system/global-pull-secret-syncer-8fcfw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal","openshift-multus/network-metrics-daemon-qhblp","openshift-network-diagnostics/network-check-target-5cmjc","openshift-ovn-kubernetes/ovnkube-node-k48sz","kube-system/konnectivity-agent-qgx5z"]
Jul 28 12:34:54.449046 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.449024    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.450454 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.450426    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.451652 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.451634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.452076 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.452055    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.452166 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.452078    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.452166 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.452090    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5kjk\""
Jul 28 12:34:54.452166 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.452080    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:34:54.453156 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.453141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.453375 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.453357    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:34:54.453375 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.453371    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:34:54.453500 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.453476    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-frd7f\""
Jul 28 12:34:54.453755 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.453716    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:34:54.454485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.454466    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.454639 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.454609    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.454639 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.454622    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.455354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455223    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:34:54.455354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455249    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.455354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455268    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fxt5v\""
Jul 28 12:34:54.455354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455315    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.455820 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455801    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:34:54.455904 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455852    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tz4zx\""
Jul 28 12:34:54.455965 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.455918    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.456044 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.456018    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.456200 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.456186    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.457285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457124    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.457285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457147    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:34:54.457285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457265    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:34:54.457464 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p7l7z\""
Jul 28 12:34:54.457801 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457562    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.457801 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457657    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:34:54.457801 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.457677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.458679 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.458630    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:34:54.458679 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.458643    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:34:54.458986 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.458954    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f5djj\""
Jul 28 12:34:54.459335 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.459316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:54.460218 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.460201    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:34:54.460629 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.460608    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.460715 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.460643    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.460862 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.460848    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bddgz\""
Jul 28 12:34:54.463015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.462500    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:34:54.463015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.462716    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:34:54.463015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.462783    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.463321 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.463299    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xwhmc\""
Jul 28 12:34:54.463642 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.463625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:54.463734 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:54.463713    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:34:54.465484 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.465465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:54.465554 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:54.465532    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:34:54.465725 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.465701    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.466409 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.466388    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.466486 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.466463    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8hrfk\""
Jul 28 12:34:54.467211 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.467194    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:54.467284 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:54.467251    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:34:54.467995 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.467914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-kubelet\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468087 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468087 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlrv\" (UniqueName: \"kubernetes.io/projected/57be213e-d569-49c8-8f8d-39e47e357423-kube-api-access-9xlrv\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.468087 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-cni-binary-copy\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-netns\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468178    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-log-socket\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-systemd-units\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-slash\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468268 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-env-overrides\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovn-node-metrics-cert\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-cnibin\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-cnibin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-netns\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-etc-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-node-log\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-multus\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468479    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-multus-certs\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468522    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-hosts-file\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-systemd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-var-lib-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-system-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-socket-dir-parent\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-bin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-hostroot\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-etc-kubernetes\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-config\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-daemon-config\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468807    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-sys-fs\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.468839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8defa946-39f1-480d-9c8c-60855d3164c0-host\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-kubelet\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748wf\" (UniqueName: \"kubernetes.io/projected/8defa946-39f1-480d-9c8c-60855d3164c0-kube-api-access-748wf\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.468948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-tmp-dir\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-ovn\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-script-lib\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-system-cni-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-netd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-os-release\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469183    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-socket-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jf6h\" (UniqueName: \"kubernetes.io/projected/48c5088c-00bb-4eba-96c8-c11269d8ae62-kube-api-access-4jf6h\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.469485 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-os-release\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-registration-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccdb8e52-a2d9-4925-857e-51be064d7650-iptables-alerter-script\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpnj\" (UniqueName: \"kubernetes.io/projected/ccdb8e52-a2d9-4925-857e-51be064d7650-kube-api-access-llpnj\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-bin\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-conf-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-k8s-cni-cncf-io\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nfc\" (UniqueName: \"kubernetes.io/projected/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-kube-api-access-n5nfc\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8defa946-39f1-480d-9c8c-60855d3164c0-serviceca\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-device-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdb8e52-a2d9-4925-857e-51be064d7650-host-slash\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zz8t\" (UniqueName: \"kubernetes.io/projected/211840ec-bc0a-42ca-9b18-7feef5d047b9-kube-api-access-4zz8t\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.470288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.469574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfz65\" (UniqueName: \"kubernetes.io/projected/ed418fcc-cb62-48f8-a809-6e8db21811fd-kube-api-access-dfz65\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.470900 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.470331    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:34:54.490591 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.490573    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9kkf"
Jul 28 12:34:54.505771 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.505745    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9kkf"
Jul 28 12:34:54.557899 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.557871    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:34:54.570193 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlrv\" (UniqueName: \"kubernetes.io/projected/57be213e-d569-49c8-8f8d-39e47e357423-kube-api-access-9xlrv\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.570193 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-cni-binary-copy\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-sys\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.570402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:54.570402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-netns\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570363    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-netns\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-log-socket\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-kubelet-config\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-systemd-units\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-slash\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570579    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-systemd-units\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-env-overrides\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovn-node-metrics-cert\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-slash\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-cnibin\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-cnibin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-log-socket\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570688    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-cnibin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-cnibin\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570753    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-cni-binary-copy\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-netns\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-kubernetes\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570815    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-netns\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-etc-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-node-log\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-multus\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-multus-certs\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-hosts-file\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-tmp\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.570956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgz6\" (UniqueName: \"kubernetes.io/projected/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-kube-api-access-lsgz6\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.570998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-systemd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-var-lib-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-env-overrides\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-system-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-var-lib-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-socket-dir-parent\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-bin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-hostroot\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571136    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-etc-kubernetes\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-bin\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-socket-dir-parent\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-hostroot\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-cni-multus\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-multus-certs\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-etc-openvswitch\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.571707 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-node-log\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-hosts-file\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-systemd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571444    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-system-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-etc-kubernetes\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-config\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-daemon-config\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-sys-fs\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-konnectivity-ca\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-lib-modules\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8defa946-39f1-480d-9c8c-60855d3164c0-host\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571891    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-dbus\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-kubelet\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.571967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748wf\" (UniqueName: \"kubernetes.io/projected/8defa946-39f1-480d-9c8c-60855d3164c0-kube-api-access-748wf\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.572534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-config\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-var-lib-kubelet\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-host\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-daemon-config\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-sys-fs\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572133    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4lq\" (UniqueName: \"kubernetes.io/projected/821352a4-2d79-4aae-836d-3c9929e25076-kube-api-access-jx4lq\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572175    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572212    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8defa946-39f1-480d-9c8c-60855d3164c0-host\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572221    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-tmp-dir\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-ovn\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-var-lib-kubelet\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-script-lib\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-run-ovn\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-system-cni-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-agent-certs\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:54.573326 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-netd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-os-release\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-tmp-dir\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-netd\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572733    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-cni-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-os-release\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-system-cni-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-socket-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jf6h\" (UniqueName: \"kubernetes.io/projected/48c5088c-00bb-4eba-96c8-c11269d8ae62-kube-api-access-4jf6h\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.572964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-os-release\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-registration-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574097 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-socket-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovnkube-script-lib\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-os-release\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccdb8e52-a2d9-4925-857e-51be064d7650-iptables-alerter-script\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpnj\" (UniqueName: \"kubernetes.io/projected/ccdb8e52-a2d9-4925-857e-51be064d7650-kube-api-access-llpnj\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-bin\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-conf-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-modprobe-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-k8s-cni-cncf-io\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nfc\" (UniqueName: \"kubernetes.io/projected/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-kube-api-access-n5nfc\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8defa946-39f1-480d-9c8c-60855d3164c0-serviceca\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-device-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysconfig\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-conf\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-systemd\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.574701 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdb8e52-a2d9-4925-857e-51be064d7650-host-slash\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-registration-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zz8t\" (UniqueName: \"kubernetes.io/projected/211840ec-bc0a-42ca-9b18-7feef5d047b9-kube-api-access-4zz8t\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfz65\" (UniqueName: \"kubernetes.io/projected/ed418fcc-cb62-48f8-a809-6e8db21811fd-kube-api-access-dfz65\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-run\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-etc-tuned\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-kubelet\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-cni-bin\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccdb8e52-a2d9-4925-857e-51be064d7650-iptables-alerter-script\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57be213e-d569-49c8-8f8d-39e47e357423-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48c5088c-00bb-4eba-96c8-c11269d8ae62-device-dir\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-multus-conf-dir\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccdb8e52-a2d9-4925-857e-51be064d7650-host-slash\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57be213e-d569-49c8-8f8d-39e47e357423-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.573966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/211840ec-bc0a-42ca-9b18-7feef5d047b9-host-kubelet\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.575331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.574004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed418fcc-cb62-48f8-a809-6e8db21811fd-host-run-k8s-cni-cncf-io\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.575871 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.574340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8defa946-39f1-480d-9c8c-60855d3164c0-serviceca\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.575871 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.575168    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/211840ec-bc0a-42ca-9b18-7feef5d047b9-ovn-node-metrics-cert\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.580260 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.580187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlrv\" (UniqueName: \"kubernetes.io/projected/57be213e-d569-49c8-8f8d-39e47e357423-kube-api-access-9xlrv\") pod \"multus-additional-cni-plugins-k6z5b\" (UID: \"57be213e-d569-49c8-8f8d-39e47e357423\") " pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:54.581213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.581189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748wf\" (UniqueName: \"kubernetes.io/projected/8defa946-39f1-480d-9c8c-60855d3164c0-kube-api-access-748wf\") pod \"node-ca-t724j\" (UID: \"8defa946-39f1-480d-9c8c-60855d3164c0\") " pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:54.582477 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.582457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zz8t\" (UniqueName: \"kubernetes.io/projected/211840ec-bc0a-42ca-9b18-7feef5d047b9-kube-api-access-4zz8t\") pod \"ovnkube-node-k48sz\" (UID: \"211840ec-bc0a-42ca-9b18-7feef5d047b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:54.583288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.583270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpnj\" (UniqueName: \"kubernetes.io/projected/ccdb8e52-a2d9-4925-857e-51be064d7650-kube-api-access-llpnj\") pod \"iptables-alerter-fw29v\" (UID: \"ccdb8e52-a2d9-4925-857e-51be064d7650\") " pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:54.588995 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.586913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nfc\" (UniqueName: \"kubernetes.io/projected/adff46d3-a8d6-44ec-bb7b-00a01a52ca78-kube-api-access-n5nfc\") pod \"node-resolver-sq4xr\" (UID: \"adff46d3-a8d6-44ec-bb7b-00a01a52ca78\") " pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:54.588995 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.587052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jf6h\" (UniqueName: \"kubernetes.io/projected/48c5088c-00bb-4eba-96c8-c11269d8ae62-kube-api-access-4jf6h\") pod \"aws-ebs-csi-driver-node-s8mwj\" (UID: \"48c5088c-00bb-4eba-96c8-c11269d8ae62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:54.588995 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.587212    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfz65\" (UniqueName: \"kubernetes.io/projected/ed418fcc-cb62-48f8-a809-6e8db21811fd-kube-api-access-dfz65\") pod \"multus-fbc2s\" (UID: \"ed418fcc-cb62-48f8-a809-6e8db21811fd\") " pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:54.665282 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:54.665260    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12610501867f6e946c0cb2390ee3cc4f.slice/crio-8a771d60206aec202c55143feb81cbb042583732107928a8f57a5b45f5102f28 WatchSource:0}: Error finding container 8a771d60206aec202c55143feb81cbb042583732107928a8f57a5b45f5102f28: Status 404 returned error can't find the container with id 8a771d60206aec202c55143feb81cbb042583732107928a8f57a5b45f5102f28
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:54.674325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-var-lib-kubelet\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-host\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4lq\" (UniqueName: \"kubernetes.io/projected/821352a4-2d79-4aae-836d-3c9929e25076-kube-api-access-jx4lq\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-agent-certs\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-var-lib-kubelet\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-host\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.102281    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-modprobe-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysconfig\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.102343    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:56.602321375 +0000 UTC m=+3.087945964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-conf\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-systemd\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-run\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-etc-tuned\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.102456 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-sys\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-conf\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102475    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-run\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysconfig\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-modprobe-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.102590    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.102636    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:56.602620083 +0000 UTC m=+3.088244675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-kubelet-config\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-kubernetes\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-sysctl-d\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-tmp\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-systemd\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-sys\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102830    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-etc-kubernetes\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgz6\" (UniqueName: \"kubernetes.io/projected/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-kube-api-access-lsgz6\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:56.103361 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-kubelet-config\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-konnectivity-ca\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-lib-modules\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.102973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-dbus\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.103048    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdff30c9-8adc-4b6b-8e8c-a771af042897-dbus\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.103100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821352a4-2d79-4aae-836d-3c9929e25076-lib-modules\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.103911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.103630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-konnectivity-ca\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:56.104876 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.104858    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-tmp\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.105075 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.105055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821352a4-2d79-4aae-836d-3c9929e25076-etc-tuned\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.105214 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.105195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13-agent-certs\") pod \"konnectivity-agent-qgx5z\" (UID: \"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13\") " pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:56.112208 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.112188    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:56.112289 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.112210    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:56.112289 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.112225    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:56.112352 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.112290    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:34:56.612273107 +0000 UTC m=+3.097897696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:56.114591 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.114570    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgz6\" (UniqueName: \"kubernetes.io/projected/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-kube-api-access-lsgz6\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:56.114848 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.114808    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4lq\" (UniqueName: \"kubernetes.io/projected/821352a4-2d79-4aae-836d-3c9929e25076-kube-api-access-jx4lq\") pod \"tuned-vfpkz\" (UID: \"821352a4-2d79-4aae-836d-3c9929e25076\") " pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.191815 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.191770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw29v"
Jul 28 12:34:56.199117 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.198972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:34:56.201462 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.201436    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdb8e52_a2d9_4925_857e_51be064d7650.slice/crio-9e298dc72106ef6c727091eecf0321c2078a0864e54ac0ffdf65f811502b22c6 WatchSource:0}: Error finding container 9e298dc72106ef6c727091eecf0321c2078a0864e54ac0ffdf65f811502b22c6: Status 404 returned error can't find the container with id 9e298dc72106ef6c727091eecf0321c2078a0864e54ac0ffdf65f811502b22c6
Jul 28 12:34:56.205971 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.205952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sq4xr"
Jul 28 12:34:56.207028 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.206999    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211840ec_bc0a_42ca_9b18_7feef5d047b9.slice/crio-18a412b448a386031582a6c21ca3d6e2d0d42fefe9338255d5318b97f755b2e7 WatchSource:0}: Error finding container 18a412b448a386031582a6c21ca3d6e2d0d42fefe9338255d5318b97f755b2e7: Status 404 returned error can't find the container with id 18a412b448a386031582a6c21ca3d6e2d0d42fefe9338255d5318b97f755b2e7
Jul 28 12:34:56.211000 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.210934    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t724j"
Jul 28 12:34:56.213792 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.213767    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadff46d3_a8d6_44ec_bb7b_00a01a52ca78.slice/crio-4bd75ada2e15d719f7c3bfba35890f434021901c49065b3f7edbeff5556d22c2 WatchSource:0}: Error finding container 4bd75ada2e15d719f7c3bfba35890f434021901c49065b3f7edbeff5556d22c2: Status 404 returned error can't find the container with id 4bd75ada2e15d719f7c3bfba35890f434021901c49065b3f7edbeff5556d22c2
Jul 28 12:34:56.218316 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.218298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6z5b"
Jul 28 12:34:56.221380 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.219594    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8defa946_39f1_480d_9c8c_60855d3164c0.slice/crio-7d2d36121c2fa9afaf1061ddf3c16b974c91ac5cdd43933efc90d8e1943ad0b1 WatchSource:0}: Error finding container 7d2d36121c2fa9afaf1061ddf3c16b974c91ac5cdd43933efc90d8e1943ad0b1: Status 404 returned error can't find the container with id 7d2d36121c2fa9afaf1061ddf3c16b974c91ac5cdd43933efc90d8e1943ad0b1
Jul 28 12:34:56.225662 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.225623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbc2s"
Jul 28 12:34:56.228031 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.228007    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57be213e_d569_49c8_8f8d_39e47e357423.slice/crio-a3319b851057729a07cb2786713ac0b4e7a3dfed9eccb30cc76bdf81ea3dc843 WatchSource:0}: Error finding container a3319b851057729a07cb2786713ac0b4e7a3dfed9eccb30cc76bdf81ea3dc843: Status 404 returned error can't find the container with id a3319b851057729a07cb2786713ac0b4e7a3dfed9eccb30cc76bdf81ea3dc843
Jul 28 12:34:56.233896 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.233876    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj"
Jul 28 12:34:56.239231 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.239216    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:34:56.241794 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.241765    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c5088c_00bb_4eba_96c8_c11269d8ae62.slice/crio-ab5b92182bbc7fd053415135f6e429b20fe7cb4c5fc504c4a1cfe4239f90870a WatchSource:0}: Error finding container ab5b92182bbc7fd053415135f6e429b20fe7cb4c5fc504c4a1cfe4239f90870a: Status 404 returned error can't find the container with id ab5b92182bbc7fd053415135f6e429b20fe7cb4c5fc504c4a1cfe4239f90870a
Jul 28 12:34:56.244110 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.244093    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfpkz"
Jul 28 12:34:56.248216 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.248180    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf2d9df_0f6f_4375_9ce5_5e803a1b1f13.slice/crio-cac96d5e700260ae38e9af5caa4ab1f51761b966c4510a4eafabfad38d590877 WatchSource:0}: Error finding container cac96d5e700260ae38e9af5caa4ab1f51761b966c4510a4eafabfad38d590877: Status 404 returned error can't find the container with id cac96d5e700260ae38e9af5caa4ab1f51761b966c4510a4eafabfad38d590877
Jul 28 12:34:56.252187 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:34:56.252164    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821352a4_2d79_4aae_836d_3c9929e25076.slice/crio-9ee91e7bd91f146c0a5658831a3e6428fa48e0c4cbd8ebcf52a8d4777b841f1d WatchSource:0}: Error finding container 9ee91e7bd91f146c0a5658831a3e6428fa48e0c4cbd8ebcf52a8d4777b841f1d: Status 404 returned error can't find the container with id 9ee91e7bd91f146c0a5658831a3e6428fa48e0c4cbd8ebcf52a8d4777b841f1d
Jul 28 12:34:56.606457 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.606377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:56.606457 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.606446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:56.606663 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.606579    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:56.606663 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.606632    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.606773 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.606638    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.606619701 +0000 UTC m=+4.092244276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:56.606773 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.606704    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.606683469 +0000 UTC m=+4.092308046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.707275 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.707223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:56.707448 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.707421    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:56.707448 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.707441    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:56.707558 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.707455    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:56.707558 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:56.707518    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.707498401 +0000 UTC m=+4.193122991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:56.936060 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.935950    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:29:55 +0000 UTC" deadline="2028-03-27 07:18:24.879875949 +0000 UTC"
Jul 28 12:34:56.936060 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:56.936012    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14586h43m27.943868218s"
Jul 28 12:34:57.036952 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.036850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"18a412b448a386031582a6c21ca3d6e2d0d42fefe9338255d5318b97f755b2e7"}
Jul 28 12:34:57.054255 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.054212    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4306b1e970225de593bcccc578a5aca" containerID="adb9c675a65b7b7ae94719f5241d149d802f82a225fdd3b198b50309d48ff1d1" exitCode=0
Jul 28 12:34:57.054435 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.054302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"c4306b1e970225de593bcccc578a5aca","Type":"ContainerDied","Data":"adb9c675a65b7b7ae94719f5241d149d802f82a225fdd3b198b50309d48ff1d1"}
Jul 28 12:34:57.058354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.058322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfpkz" event={"ID":"821352a4-2d79-4aae-836d-3c9929e25076","Type":"ContainerStarted","Data":"9ee91e7bd91f146c0a5658831a3e6428fa48e0c4cbd8ebcf52a8d4777b841f1d"}
Jul 28 12:34:57.069164 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.069136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qgx5z" event={"ID":"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13","Type":"ContainerStarted","Data":"cac96d5e700260ae38e9af5caa4ab1f51761b966c4510a4eafabfad38d590877"}
Jul 28 12:34:57.084704 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.084674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbc2s" event={"ID":"ed418fcc-cb62-48f8-a809-6e8db21811fd","Type":"ContainerStarted","Data":"17448253bcd4eb4d9cb101f827c64172a2c5449cbc851e5437c312feafff6b43"}
Jul 28 12:34:57.106285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.106002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerStarted","Data":"a3319b851057729a07cb2786713ac0b4e7a3dfed9eccb30cc76bdf81ea3dc843"}
Jul 28 12:34:57.118663 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.118626    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sq4xr" event={"ID":"adff46d3-a8d6-44ec-bb7b-00a01a52ca78","Type":"ContainerStarted","Data":"4bd75ada2e15d719f7c3bfba35890f434021901c49065b3f7edbeff5556d22c2"}
Jul 28 12:34:57.135511 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.135426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw29v" event={"ID":"ccdb8e52-a2d9-4925-857e-51be064d7650","Type":"ContainerStarted","Data":"9e298dc72106ef6c727091eecf0321c2078a0864e54ac0ffdf65f811502b22c6"}
Jul 28 12:34:57.140153 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.140113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" event={"ID":"12610501867f6e946c0cb2390ee3cc4f","Type":"ContainerStarted","Data":"8a771d60206aec202c55143feb81cbb042583732107928a8f57a5b45f5102f28"}
Jul 28 12:34:57.148308 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.148276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj" event={"ID":"48c5088c-00bb-4eba-96c8-c11269d8ae62","Type":"ContainerStarted","Data":"ab5b92182bbc7fd053415135f6e429b20fe7cb4c5fc504c4a1cfe4239f90870a"}
Jul 28 12:34:57.155224 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.155195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t724j" event={"ID":"8defa946-39f1-480d-9c8c-60855d3164c0","Type":"ContainerStarted","Data":"7d2d36121c2fa9afaf1061ddf3c16b974c91ac5cdd43933efc90d8e1943ad0b1"}
Jul 28 12:34:57.614483 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.614373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:57.614483 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.614451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:57.614714 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.614548    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:57.614714 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.614577    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:57.614714 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.614630    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:59.614610161 +0000 UTC m=+6.100234755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:57.614714 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.614650    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:59.614640367 +0000 UTC m=+6.100264942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:57.718109 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.718074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:57.718274 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.718245    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:57.718274 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.718267    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:57.718387 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.718280    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:57.718387 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:57.718338    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:34:59.718319872 +0000 UTC m=+6.203944456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:57.936602 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.936478    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:29:55 +0000 UTC" deadline="2028-02-16 16:12:33.673852288 +0000 UTC"
Jul 28 12:34:57.936602 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:57.936520    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13635h37m35.737337103s"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:58.014708    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:58.014865    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:58.015238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:58.015319    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:58.015379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:58.015498 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:58.015445    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:34:58.172408 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:58.172353    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"c4306b1e970225de593bcccc578a5aca","Type":"ContainerStarted","Data":"b4f90ffbac8c0059c10905186cbe08334f6bca0891f8e96888cd77c90770f6c4"}
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:59.634390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:59.634447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.634548    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.634557    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.634604    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:03.634591984 +0000 UTC m=+10.120216554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:59.634638 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.634618    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:03.634612213 +0000 UTC m=+10.120236783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:59.735117 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:34:59.735077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:34:59.735302 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.735279    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:59.735302 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.735301    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:59.735413 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.735314    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:59.735413 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:34:59.735368    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:03.73535154 +0000 UTC m=+10.220976127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:00.012916 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:00.012825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:00.013054 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:00.012922    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:00.013054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:00.012836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:00.013054 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:00.012991    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:00.013054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:00.012825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:00.013054 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:00.013052    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:02.012497 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:02.012460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:02.012992 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:02.012460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:02.012992 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:02.012592    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:02.012992 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:02.012693    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:02.012992 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:02.012463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:02.012992 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:02.012801    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:03.666979 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:03.666937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:03.667425 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:03.667009    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:03.667425 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.667136    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:03.667425 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.667138    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:03.667425 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.667207    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.667186912 +0000 UTC m=+18.152811498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:03.667425 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.667227    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.667217251 +0000 UTC m=+18.152841821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:03.768854 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:03.768152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:03.768854 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.768323    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:03.768854 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.768341    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:03.768854 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.768354    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:03.768854 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:03.768461    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.768441757 +0000 UTC m=+18.254066340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:04.013094    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:04.013174    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:04.013537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:04.013648    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:04.013696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:04.013881 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:04.013790    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:06.011839    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:06.011887    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:06.011837    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:06.011974    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:06.012062    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:06.012340 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:06.012127    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:08.012131 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:08.012096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:08.012525 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:08.012096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:08.012525 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:08.012226    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:08.012525 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:08.012096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:08.012525 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:08.012305    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:08.012525 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:08.012389    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:10.012450 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:10.012407    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:10.012450 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:10.012458    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:10.013000 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:10.012566    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:10.013000 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:10.012630    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:10.013000 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:10.012709    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:10.013000 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:10.012803    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:11.725632 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:11.725591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:11.726144 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:11.725655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:11.726144 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.725777    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.726144 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.725856    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.725834655 +0000 UTC m=+34.211459238 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.726144 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.725776    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.726144 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.725936    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.725922175 +0000 UTC m=+34.211546745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.826686 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:11.826650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:11.826887 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.826824    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:11.826887 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.826847    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:11.826887 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.826858    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:11.827045 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:11.826921    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.826903201 +0000 UTC m=+34.312527794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:12.011860 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:12.011775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:12.012082 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:12.011776    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:12.012082 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:12.011914    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:12.012082 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:12.012008    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:12.012082 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:12.011776    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:12.012272 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:12.012106    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:14.012746 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:14.012498    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:14.012746 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:14.012613    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:14.012746 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:14.012623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:14.012746 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:14.012700    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:14.012746 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:14.012714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:14.013393 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:14.012848    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:15.208532 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.208494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfpkz" event={"ID":"821352a4-2d79-4aae-836d-3c9929e25076","Type":"ContainerStarted","Data":"a6b9aa0464f6eec91a6b3eef99f2c576e800cf8fee29afbf719ae30842bfa6ba"}
Jul 28 12:35:15.210181 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.210156    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerStarted","Data":"464ae24fb19572391e002289d4cd558cad37893335b5025a3d58e5954dd43543"}
Jul 28 12:35:15.212292 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.212177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sq4xr" event={"ID":"adff46d3-a8d6-44ec-bb7b-00a01a52ca78","Type":"ContainerStarted","Data":"8676185c440285b85c73ef2baadb9854cca4edf89149b43968568b19c5109e39"}
Jul 28 12:35:15.213851 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.213824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" event={"ID":"12610501867f6e946c0cb2390ee3cc4f","Type":"ContainerStarted","Data":"e0b92ade6cf000c7ea3f3629ee003632130b4fde808910b02bfb4e7fe79b5333"}
Jul 28 12:35:15.215261 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.215243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj" event={"ID":"48c5088c-00bb-4eba-96c8-c11269d8ae62","Type":"ContainerStarted","Data":"92e19cd2110430751297a51733a4163a499e7290e31b1cd175af1f056f88549d"}
Jul 28 12:35:15.216672 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.216652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t724j" event={"ID":"8defa946-39f1-480d-9c8c-60855d3164c0","Type":"ContainerStarted","Data":"4a50c422d76471ddc3fec50a78a47fe69fadc691986bbc3e2ca25f2711b896cc"}
Jul 28 12:35:15.229937 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.229760    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" podStartSLOduration=21.22974893 podStartE2EDuration="21.22974893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:58.189182963 +0000 UTC m=+4.674807559" watchObservedRunningTime="2026-07-28 12:35:15.22974893 +0000 UTC m=+21.715373522"
Jul 28 12:35:15.230182 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.230162    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vfpkz" podStartSLOduration=6.6570403989999996 podStartE2EDuration="21.230157384s" totalImagesPullingTime="14.573116985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.229577243 +0000 UTC m=+21.715201835" watchObservedRunningTime="2026-07-28 12:35:15.230157384 +0000 UTC m=+21.715781976"
Jul 28 12:35:15.266322 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.266266    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" podStartSLOduration=20.266247944 podStartE2EDuration="20.266247944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.248154661 +0000 UTC m=+21.733779253" watchObservedRunningTime="2026-07-28 12:35:15.266247944 +0000 UTC m=+21.751872538"
Jul 28 12:35:15.266534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.266507    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sq4xr" podStartSLOduration=2.508407771 podStartE2EDuration="21.266495951s" totalImagesPullingTime="18.75808818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.265695362 +0000 UTC m=+21.751319951" watchObservedRunningTime="2026-07-28 12:35:15.266495951 +0000 UTC m=+21.752120547"
Jul 28 12:35:15.303849 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:15.303807    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t724j" podStartSLOduration=6.698866109 podStartE2EDuration="21.303796407s" totalImagesPullingTime="14.604930298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.303631487 +0000 UTC m=+21.789256078" watchObservedRunningTime="2026-07-28 12:35:15.303796407 +0000 UTC m=+21.789420998"
Jul 28 12:35:16.012626 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.012453    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:16.012775 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.012454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:16.012775 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:16.012743    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:16.012884 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.012501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:16.012884 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:16.012820    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:16.012987 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:16.012880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:16.222979 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.222422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qgx5z" event={"ID":"bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13","Type":"ContainerStarted","Data":"e1c26f493087dc6ce56251b8c418db277a9f59f9e2e0f6ccc2e56658daa3b6e6"}
Jul 28 12:35:16.224847 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.224824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbc2s" event={"ID":"ed418fcc-cb62-48f8-a809-6e8db21811fd","Type":"ContainerStarted","Data":"892b81f227c512894df26f98c37c4a554afcd418b0c30e332238034ad1fb45c8"}
Jul 28 12:35:16.226181 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.226160    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="464ae24fb19572391e002289d4cd558cad37893335b5025a3d58e5954dd43543" exitCode=0
Jul 28 12:35:16.226285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.226243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"464ae24fb19572391e002289d4cd558cad37893335b5025a3d58e5954dd43543"}
Jul 28 12:35:16.228150 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228134    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:35:16.228403 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228387    2558 generic.go:356] "Generic (PLEG): container finished" podID="211840ec-bc0a-42ca-9b18-7feef5d047b9" containerID="006387ca8993b9e899759f95857f2d60a322c89e8292eb8f094f05ba1a6c011e" exitCode=1
Jul 28 12:35:16.228482 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"943bfbde96f79366ef59ea90a72be1aa7642a569c30bf17b73d45566d0ccff68"}
Jul 28 12:35:16.228555 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"30ad901d6a3abe269b0942cdae37b4eb22983d7e37cfe022d7cd54d87cf9b68d"}
Jul 28 12:35:16.228612 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerDied","Data":"006387ca8993b9e899759f95857f2d60a322c89e8292eb8f094f05ba1a6c011e"}
Jul 28 12:35:16.228663 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.228652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"5d601d63b3d230b0bf28119209aa31dc1b5e028eae0c14b78adb166c94f23094"}
Jul 28 12:35:16.246180 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.246142    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qgx5z" podStartSLOduration=3.498731369 podStartE2EDuration="22.246126714s" totalImagesPullingTime="18.747395345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:16.245690055 +0000 UTC m=+22.731314647" watchObservedRunningTime="2026-07-28 12:35:16.246126714 +0000 UTC m=+22.731751305"
Jul 28 12:35:16.290120 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.290079    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fbc2s" podStartSLOduration=3.516120864 podStartE2EDuration="22.290064048s" totalImagesPullingTime="18.773943184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:16.290043298 +0000 UTC m=+22.775667890" watchObservedRunningTime="2026-07-28 12:35:16.290064048 +0000 UTC m=+22.775688639"
Jul 28 12:35:16.423441 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.423418    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:35:16.957864 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.957749    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:35:16.423436594Z","UUID":"c09eece9-2518-40f0-8b2c-1939d623ae8f","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:35:16.959892 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.959866    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:35:16.960003 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:16.959908    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:35:17.233144 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.233052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw29v" event={"ID":"ccdb8e52-a2d9-4925-857e-51be064d7650","Type":"ContainerStarted","Data":"4c5cf67e4a90171951f3b267d28e40a31bdad4c3d6e036e0e6b2ea227e93dbc5"}
Jul 28 12:35:17.235754 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.235705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj" event={"ID":"48c5088c-00bb-4eba-96c8-c11269d8ae62","Type":"ContainerStarted","Data":"90422c24f844cf56f940a36c5ee94ad75b3d09fe4868d9f90f881f3068de6818"}
Jul 28 12:35:17.239377 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.239355    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:35:17.240238 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.239761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"7024969fd5d4b9f2b830660679f8dcf2ffcd47e4fc479f8f40022528afd44f8b"}
Jul 28 12:35:17.240238 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.239838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"c822e20f6f42daa50a151bccba9af7806853ee7360cab933b06b668bfc24f721"}
Jul 28 12:35:17.250878 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:17.250826    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fw29v" podStartSLOduration=4.462952315 podStartE2EDuration="23.250807035s" totalImagesPullingTime="18.78785472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:17.250433853 +0000 UTC m=+23.736058446" watchObservedRunningTime="2026-07-28 12:35:17.250807035 +0000 UTC m=+23.736431628"
Jul 28 12:35:18.012180 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:18.011996    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:18.012448 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:18.011997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:18.012448 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:18.012263    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:18.012448 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:18.011997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:18.012448 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:18.012347    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:18.012448 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:18.012434    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:18.245010 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:18.244972    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj" event={"ID":"48c5088c-00bb-4eba-96c8-c11269d8ae62","Type":"ContainerStarted","Data":"da1194b43dbd21d996938cacc316e5ad2ead690bb97a1864f05e40492b5223a9"}
Jul 28 12:35:18.265773 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:18.265672    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8mwj" podStartSLOduration=3.5938627199999997 podStartE2EDuration="24.265656271s" totalImagesPullingTime="20.671793551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.265341832 +0000 UTC m=+24.750966423" watchObservedRunningTime="2026-07-28 12:35:18.265656271 +0000 UTC m=+24.751280893"
Jul 28 12:35:19.250577 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:19.250545    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:35:19.251115 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:19.250982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"fd4921bcfb289b7488a0ba8e292145b2cb7a804837e00df25838e30f4fd61c06"}
Jul 28 12:35:20.012499 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.012470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:20.012680 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.012470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:20.012680 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:20.012624    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:20.012680 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.012480    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:20.012908 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:20.012672    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:20.012908 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:20.012779    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:20.082232 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.082197    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:35:20.082864 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.082841    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:35:20.253630 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.253604    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:35:20.254177 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:20.254001    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qgx5z"
Jul 28 12:35:21.257843 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.257644    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="6fccc4e62f14cb29f10256b17d7e7497be45e4a4006e9ec05a94ee447015cb74" exitCode=0
Jul 28 12:35:21.258465 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.257742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"6fccc4e62f14cb29f10256b17d7e7497be45e4a4006e9ec05a94ee447015cb74"}
Jul 28 12:35:21.261281 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.261263    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:35:21.261639 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.261615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"f94d414114fca5e062b80fa1b1f1cf72857aeeae60564d2a0d667af35758dead"}
Jul 28 12:35:21.261902 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.261880    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:21.262147 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.261948    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:21.262147 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.261997    2558 scope.go:117] "RemoveContainer" containerID="006387ca8993b9e899759f95857f2d60a322c89e8292eb8f094f05ba1a6c011e"
Jul 28 12:35:21.278285 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.278264    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:21.278386 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:21.278376    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.013863    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.014012    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.014418    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.014511    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.014628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:22.015742 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.014735    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:22.267270 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.267211    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:35:22.267758 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.267590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" event={"ID":"211840ec-bc0a-42ca-9b18-7feef5d047b9","Type":"ContainerStarted","Data":"266ed32bcbdb4dd47d8fd8c9e26425a0e590ac12057ba4ba656a34c9b486e587"}
Jul 28 12:35:22.267758 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.267609    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 12:35:22.302713 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.302666    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" podStartSLOduration=8.866816495 podStartE2EDuration="28.302650411s" totalImagesPullingTime="19.435833916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:22.302431411 +0000 UTC m=+28.788056004" watchObservedRunningTime="2026-07-28 12:35:22.302650411 +0000 UTC m=+28.788275002"
Jul 28 12:35:22.792555 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.792346    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cmjc"]
Jul 28 12:35:22.792779 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.792585    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:22.792779 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.792673    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:22.802793 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.802761    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhblp"]
Jul 28 12:35:22.802911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.802874    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:22.802978 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.802962    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:22.825092 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.825059    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fcfw"]
Jul 28 12:35:22.825218 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:22.825168    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:22.825286 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:22.825269    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:23.271732 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:23.271679    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="872bde41b43c5482c0ebbcc3ed2f4fb0a30d013aee52c2ae6a639a5bcedb08e5" exitCode=0
Jul 28 12:35:23.272302 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:23.271754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"872bde41b43c5482c0ebbcc3ed2f4fb0a30d013aee52c2ae6a639a5bcedb08e5"}
Jul 28 12:35:23.272302 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:23.271842    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 12:35:24.013333 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:24.013302    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:24.013488 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:24.013400    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:25.012047 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.012019    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:25.012612 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:25.012109    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:25.012612 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.012169    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:25.012612 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:25.012268    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:25.278814 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.278783    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="cb56fc26c11065133dba0fc14a3ec74f5352e4344b3bb4079ddb5ceb95e0a038" exitCode=0
Jul 28 12:35:25.278975 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.278840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"cb56fc26c11065133dba0fc14a3ec74f5352e4344b3bb4079ddb5ceb95e0a038"}
Jul 28 12:35:25.306505 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.306479    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:25.306689 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.306675    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 12:35:25.323211 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.323175    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" podUID="211840ec-bc0a-42ca-9b18-7feef5d047b9" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 28 12:35:25.332141 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:25.332114    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz" podUID="211840ec-bc0a-42ca-9b18-7feef5d047b9" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 28 12:35:26.012054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:26.012028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:26.012688 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:26.012241    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:27.012534 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.012500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:27.012964 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.012500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:27.012964 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.012610    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fcfw" podUID="cdff30c9-8adc-4b6b-8e8c-a771af042897"
Jul 28 12:35:27.012964 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.012744    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cmjc" podUID="611e17b6-2857-4c4d-93e1-77d997af210d"
Jul 28 12:35:27.206792 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.206757    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sq4xr_adff46d3-a8d6-44ec-bb7b-00a01a52ca78/dns-node-resolver/0.log"
Jul 28 12:35:27.739539 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.739510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:27.739688 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.739564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:27.739688 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.739644    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:27.739688 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.739645    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:27.739819 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.739707    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret podName:cdff30c9-8adc-4b6b-8e8c-a771af042897 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:59.739687833 +0000 UTC m=+66.225312404 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret") pod "global-pull-secret-syncer-8fcfw" (UID: "cdff30c9-8adc-4b6b-8e8c-a771af042897") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:27.739819 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.739742    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs podName:911090a5-d1b2-49eb-ab42-3a239ccaa4f0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:59.739714873 +0000 UTC m=+66.225339443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs") pod "network-metrics-daemon-qhblp" (UID: "911090a5-d1b2-49eb-ab42-3a239ccaa4f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:27.840167 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.840083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:27.840320 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.840248    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:27.840320 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.840272    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:27.840320 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.840283    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rp69p for pod openshift-network-diagnostics/network-check-target-5cmjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:27.840472 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:27.840345    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p podName:611e17b6-2857-4c4d-93e1-77d997af210d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:59.840325077 +0000 UTC m=+66.325949661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rp69p" (UniqueName: "kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p") pod "network-check-target-5cmjc" (UID: "611e17b6-2857-4c4d-93e1-77d997af210d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:27.977224 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:27.977197    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t724j_8defa946-39f1-480d-9c8c-60855d3164c0/node-ca/0.log"
Jul 28 12:35:28.012648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.012592    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:28.013130 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:28.012767    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhblp" podUID="911090a5-d1b2-49eb-ab42-3a239ccaa4f0"
Jul 28 12:35:28.846983 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.846750    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeReady"
Jul 28 12:35:28.847145 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.847090    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:35:28.909854 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.909816    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lfppf"]
Jul 28 12:35:28.912993 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.912972    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m9zjd"]
Jul 28 12:35:28.913111 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.913089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:28.914958 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.914940    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:28.915653 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.915633    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:35:28.916018 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.915787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:35:28.916018 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.915946    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:35:28.916650 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.916177    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:35:28.916650 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.916203    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-64vgn\""
Jul 28 12:35:28.917311 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.917290    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:35:28.917969 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.917944    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:35:28.918258 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.918059    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:35:28.918258 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.918090    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s2w2r\""
Jul 28 12:35:28.928224 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.928199    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfppf"]
Jul 28 12:35:28.932119 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.931335    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q4c6v"]
Jul 28 12:35:28.933306 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.933284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:28.938579 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.938556    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:35:28.938914 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.938686    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:35:28.938914 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.938693    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4g9qb\""
Jul 28 12:35:28.947687 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.947668    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m9zjd"]
Jul 28 12:35:28.956547 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:28.953141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q4c6v"]
Jul 28 12:35:29.012196 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.012084    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:29.012196 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.012110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:29.017501 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.017478    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:35:29.017907 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.017522    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:35:29.017907 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.017482    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:35:29.017907 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.017591    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc56m\""
Jul 28 12:35:29.047473 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d16b4e-aa12-40c5-a466-2202e6a14ff3-data-volume\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49wz\" (UniqueName: \"kubernetes.io/projected/cc234e4e-0c55-4983-bbb0-e129c1b63107-kube-api-access-m49wz\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90d16b4e-aa12-40c5-a466-2202e6a14ff3-crio-socket\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8d8022-51f4-425c-bc23-3023a521481f-config-volume\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.047868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlh62\" (UniqueName: \"kubernetes.io/projected/1f8d8022-51f4-425c-bc23-3023a521481f-kube-api-access-vlh62\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.047868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc234e4e-0c55-4983-bbb0-e129c1b63107-cert\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.047868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.047868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047752    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d8022-51f4-425c-bc23-3023a521481f-tmp-dir\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.047868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.047780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjv75\" (UniqueName: \"kubernetes.io/projected/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-api-access-xjv75\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148611 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d8022-51f4-425c-bc23-3023a521481f-tmp-dir\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjv75\" (UniqueName: \"kubernetes.io/projected/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-api-access-xjv75\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d16b4e-aa12-40c5-a466-2202e6a14ff3-data-volume\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.148741    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:29.148790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148789    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49wz\" (UniqueName: \"kubernetes.io/projected/cc234e4e-0c55-4983-bbb0-e129c1b63107-kube-api-access-m49wz\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.148832    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:29.648812527 +0000 UTC m=+36.134437097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90d16b4e-aa12-40c5-a466-2202e6a14ff3-crio-socket\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8d8022-51f4-425c-bc23-3023a521481f-config-volume\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlh62\" (UniqueName: \"kubernetes.io/projected/1f8d8022-51f4-425c-bc23-3023a521481f-kube-api-access-vlh62\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.148990 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.148942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc234e4e-0c55-4983-bbb0-e129c1b63107-cert\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.149165 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.149003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d8022-51f4-425c-bc23-3023a521481f-tmp-dir\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.149165 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.149072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d16b4e-aa12-40c5-a466-2202e6a14ff3-data-volume\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.149165 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.149080    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:29.149165 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.149109    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:29.149165 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.149144    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls podName:1f8d8022-51f4-425c-bc23-3023a521481f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:29.649130855 +0000 UTC m=+36.134755425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls") pod "dns-default-q4c6v" (UID: "1f8d8022-51f4-425c-bc23-3023a521481f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:29.149378 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.149168    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:29.649151874 +0000 UTC m=+36.134776447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:29.149378 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.149185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90d16b4e-aa12-40c5-a466-2202e6a14ff3-crio-socket\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.149641 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.149620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8d8022-51f4-425c-bc23-3023a521481f-config-volume\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.153595 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.153573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc234e4e-0c55-4983-bbb0-e129c1b63107-cert\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.163034 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.163005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjv75\" (UniqueName: \"kubernetes.io/projected/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-api-access-xjv75\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.163332 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.163310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49wz\" (UniqueName: \"kubernetes.io/projected/cc234e4e-0c55-4983-bbb0-e129c1b63107-kube-api-access-m49wz\") pod \"ingress-canary-m9zjd\" (UID: \"cc234e4e-0c55-4983-bbb0-e129c1b63107\") " pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.165545 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.165528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlh62\" (UniqueName: \"kubernetes.io/projected/1f8d8022-51f4-425c-bc23-3023a521481f-kube-api-access-vlh62\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.230854 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.230835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m9zjd"
Jul 28 12:35:29.377640 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.377607    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m9zjd"]
Jul 28 12:35:29.380988 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:29.380952    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc234e4e_0c55_4983_bbb0_e129c1b63107.slice/crio-2205b1c29bb71a2460854e79f4ecbd30bee4aaa728ef192c35a10046955960d3 WatchSource:0}: Error finding container 2205b1c29bb71a2460854e79f4ecbd30bee4aaa728ef192c35a10046955960d3: Status 404 returned error can't find the container with id 2205b1c29bb71a2460854e79f4ecbd30bee4aaa728ef192c35a10046955960d3
Jul 28 12:35:29.652489 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.652444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:29.652489 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.652503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652593    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652610    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652639    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:30.652626713 +0000 UTC m=+37.138251286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:29.652637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652672    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls podName:1f8d8022-51f4-425c-bc23-3023a521481f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:30.652657503 +0000 UTC m=+37.138282100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls") pod "dns-default-q4c6v" (UID: "1f8d8022-51f4-425c-bc23-3023a521481f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652698    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:29.652786 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:29.652769    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:30.652756568 +0000 UTC m=+37.138381139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:30.012883 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.012629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:30.015741 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.015698    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf2p5\""
Jul 28 12:35:30.015874 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.015832    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:35:30.291115 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.291026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m9zjd" event={"ID":"cc234e4e-0c55-4983-bbb0-e129c1b63107","Type":"ContainerStarted","Data":"2205b1c29bb71a2460854e79f4ecbd30bee4aaa728ef192c35a10046955960d3"}
Jul 28 12:35:30.294230 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.294200    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="79e7dfb28b91568590e0bec4d3be0824a27ff259f3dcf6e740793516ca5763a3" exitCode=0
Jul 28 12:35:30.294380 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.294251    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"79e7dfb28b91568590e0bec4d3be0824a27ff259f3dcf6e740793516ca5763a3"}
Jul 28 12:35:30.593757 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.593656    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"]
Jul 28 12:35:30.595942 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.595911    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"
Jul 28 12:35:30.599046 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.598787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w75kh\""
Jul 28 12:35:30.599046 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.598850    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:30.599218 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.599108    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 12:35:30.605041 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.605014    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"]
Jul 28 12:35:30.662207 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.662173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:30.662365 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.662236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:30.662365 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.662274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z759g\" (UniqueName: \"kubernetes.io/projected/d6979445-3b85-4f61-a570-b21e8b788ffd-kube-api-access-z759g\") pod \"migrator-6865c9967-bt9cn\" (UID: \"d6979445-3b85-4f61-a570-b21e8b788ffd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"
Jul 28 12:35:30.662365 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.662303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:30.662365 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662345    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:30.662560 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662398    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:30.662560 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662428    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls podName:1f8d8022-51f4-425c-bc23-3023a521481f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:32.662403631 +0000 UTC m=+39.148028206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls") pod "dns-default-q4c6v" (UID: "1f8d8022-51f4-425c-bc23-3023a521481f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:30.662560 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662479    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:32.662461828 +0000 UTC m=+39.148086402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:30.662560 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662480    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:30.662560 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:30.662528    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:32.662514569 +0000 UTC m=+39.148139151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:30.763320 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.763286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z759g\" (UniqueName: \"kubernetes.io/projected/d6979445-3b85-4f61-a570-b21e8b788ffd-kube-api-access-z759g\") pod \"migrator-6865c9967-bt9cn\" (UID: \"d6979445-3b85-4f61-a570-b21e8b788ffd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"
Jul 28 12:35:30.773008 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.772975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z759g\" (UniqueName: \"kubernetes.io/projected/d6979445-3b85-4f61-a570-b21e8b788ffd-kube-api-access-z759g\") pod \"migrator-6865c9967-bt9cn\" (UID: \"d6979445-3b85-4f61-a570-b21e8b788ffd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"
Jul 28 12:35:30.906458 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:30.906379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"
Jul 28 12:35:31.226789 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.226697    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:35:31.229412 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.229378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.232128 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.232101    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:35:31.232851 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.232828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:35:31.232952 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.232877    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:35:31.233137 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.233112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gnxj\""
Jul 28 12:35:31.239901 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.239880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:35:31.246962 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.246933    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:35:31.300523 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.300495    2558 generic.go:356] "Generic (PLEG): container finished" podID="57be213e-d569-49c8-8f8d-39e47e357423" containerID="9cff66b3d9fbc4449d046a50cd5d5295cdbf2cb6b2b638bc0b72a4bd489feb03" exitCode=0
Jul 28 12:35:31.301057 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.300557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerDied","Data":"9cff66b3d9fbc4449d046a50cd5d5295cdbf2cb6b2b638bc0b72a4bd489feb03"}
Jul 28 12:35:31.368752 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.368700    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.368940 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.368895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369001 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.368946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369001 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.368968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369100 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.369070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369158 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.369109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5q4\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369209 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.369182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.369311 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.369292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.469979 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.469942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.469979 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.469978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.470096    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.470121    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-964f6bf67-wct99: secret "image-registry-tls" not found
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.470183    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls podName:56de39ce-f39c-4563-9746-e630fb800d6a nodeName:}" failed. No retries permitted until 2026-07-28 12:35:31.970160959 +0000 UTC m=+38.455785560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls") pod "image-registry-964f6bf67-wct99" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a") : secret "image-registry-tls" not found
Jul 28 12:35:31.470207 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470468 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470468 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470468 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5q4\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.470468 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.470392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.471295 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.471260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.471927 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.471907    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.473443 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.473419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.473535 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.473423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.483859 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.483802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5q4\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.486132 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.485084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.738026 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.737831    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn"]
Jul 28 12:35:31.739827 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:31.739805    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6979445_3b85_4f61_a570_b21e8b788ffd.slice/crio-ae295027624213bb62950e6c7eab50b89acbc3229ce118d04d8c1d73ddf9d946 WatchSource:0}: Error finding container ae295027624213bb62950e6c7eab50b89acbc3229ce118d04d8c1d73ddf9d946: Status 404 returned error can't find the container with id ae295027624213bb62950e6c7eab50b89acbc3229ce118d04d8c1d73ddf9d946
Jul 28 12:35:31.974447 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:31.974415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:31.974608 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.974551    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:31.974608 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.974563    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-964f6bf67-wct99: secret "image-registry-tls" not found
Jul 28 12:35:31.974684 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:31.974614    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls podName:56de39ce-f39c-4563-9746-e630fb800d6a nodeName:}" failed. No retries permitted until 2026-07-28 12:35:32.974599603 +0000 UTC m=+39.460224194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls") pod "image-registry-964f6bf67-wct99" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a") : secret "image-registry-tls" not found
Jul 28 12:35:32.304853 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.304817    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m9zjd" event={"ID":"cc234e4e-0c55-4983-bbb0-e129c1b63107","Type":"ContainerStarted","Data":"53e7c42840b2069d16c528d66a6b98d6215dd83ba17d7d075fe3bbd8b647650a"}
Jul 28 12:35:32.305957 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.305931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn" event={"ID":"d6979445-3b85-4f61-a570-b21e8b788ffd","Type":"ContainerStarted","Data":"ae295027624213bb62950e6c7eab50b89acbc3229ce118d04d8c1d73ddf9d946"}
Jul 28 12:35:32.309039 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.309018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" event={"ID":"57be213e-d569-49c8-8f8d-39e47e357423","Type":"ContainerStarted","Data":"08a4f388cca7d9a46580f92e800571837a728b733fe9e55922f5876cb5c2e094"}
Jul 28 12:35:32.321708 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.321666    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m9zjd" podStartSLOduration=1.978799207 podStartE2EDuration="4.321650591s" totalImagesPullingTime="2.342851384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:32.320830465 +0000 UTC m=+38.806455057" watchObservedRunningTime="2026-07-28 12:35:32.321650591 +0000 UTC m=+38.807275226"
Jul 28 12:35:32.346759 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.346695    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k6z5b" podStartSLOduration=9.191846698 podStartE2EDuration="38.346679981s" totalImagesPullingTime="29.154833283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:32.346164398 +0000 UTC m=+38.831788990" watchObservedRunningTime="2026-07-28 12:35:32.346679981 +0000 UTC m=+38.832304573"
Jul 28 12:35:32.510648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.510615    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vxhgw"]
Jul 28 12:35:32.536286 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.536259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vxhgw"]
Jul 28 12:35:32.536435 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.536352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.539877 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.539854    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bvq7r\""
Jul 28 12:35:32.539877 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.539876    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 12:35:32.540071 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.539906    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 12:35:32.540071 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.539854    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 12:35:32.540071 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.539986    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 12:35:32.681753 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.681707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:32.681916 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.681769    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/830400de-c238-479c-9bbe-f1cab97feaa5-signing-key\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.681916 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.681803    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:32.681916 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.681818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/830400de-c238-479c-9bbe-f1cab97feaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.681916 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.681859    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:32.681916 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.681877    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:36.681856401 +0000 UTC m=+43.167480971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.681944    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.681958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.681986    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls podName:1f8d8022-51f4-425c-bc23-3023a521481f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:36.681973033 +0000 UTC m=+43.167597607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls") pod "dns-default-q4c6v" (UID: "1f8d8022-51f4-425c-bc23-3023a521481f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.682035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdm5\" (UniqueName: \"kubernetes.io/projected/830400de-c238-479c-9bbe-f1cab97feaa5-kube-api-access-rwdm5\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.682043    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:32.682162 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.682083    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:36.682070391 +0000 UTC m=+43.167694965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:32.688830 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.688804    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"]
Jul 28 12:35:32.714261 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.714236    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"]
Jul 28 12:35:32.714424 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.714362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"
Jul 28 12:35:32.717910 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.717895    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-428ww\""
Jul 28 12:35:32.782468 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.782436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k72fz\" (UniqueName: \"kubernetes.io/projected/eddedb54-cc14-4cb7-ac99-4683df900722-kube-api-access-k72fz\") pod \"network-check-source-7d88cb55d8-dq99c\" (UID: \"eddedb54-cc14-4cb7-ac99-4683df900722\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"
Jul 28 12:35:32.782638 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.782507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdm5\" (UniqueName: \"kubernetes.io/projected/830400de-c238-479c-9bbe-f1cab97feaa5-kube-api-access-rwdm5\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.782638 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.782534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/830400de-c238-479c-9bbe-f1cab97feaa5-signing-key\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.782638 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.782575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/830400de-c238-479c-9bbe-f1cab97feaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.783156 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.783139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/830400de-c238-479c-9bbe-f1cab97feaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.797123 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.797095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/830400de-c238-479c-9bbe-f1cab97feaa5-signing-key\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.798947 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.798925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdm5\" (UniqueName: \"kubernetes.io/projected/830400de-c238-479c-9bbe-f1cab97feaa5-kube-api-access-rwdm5\") pod \"service-ca-658d7b7c78-vxhgw\" (UID: \"830400de-c238-479c-9bbe-f1cab97feaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.845368 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.845335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw"
Jul 28 12:35:32.883536 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.883509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k72fz\" (UniqueName: \"kubernetes.io/projected/eddedb54-cc14-4cb7-ac99-4683df900722-kube-api-access-k72fz\") pod \"network-check-source-7d88cb55d8-dq99c\" (UID: \"eddedb54-cc14-4cb7-ac99-4683df900722\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"
Jul 28 12:35:32.892792 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.892760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k72fz\" (UniqueName: \"kubernetes.io/projected/eddedb54-cc14-4cb7-ac99-4683df900722-kube-api-access-k72fz\") pod \"network-check-source-7d88cb55d8-dq99c\" (UID: \"eddedb54-cc14-4cb7-ac99-4683df900722\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"
Jul 28 12:35:32.984225 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.984054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:32.984393 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.984200    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:32.984393 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.984282    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-964f6bf67-wct99: secret "image-registry-tls" not found
Jul 28 12:35:32.984393 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:32.984359    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls podName:56de39ce-f39c-4563-9746-e630fb800d6a nodeName:}" failed. No retries permitted until 2026-07-28 12:35:34.984339813 +0000 UTC m=+41.469964401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls") pod "image-registry-964f6bf67-wct99" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a") : secret "image-registry-tls" not found
Jul 28 12:35:32.984887 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:32.984859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vxhgw"]
Jul 28 12:35:32.987379 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:32.987354    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830400de_c238_479c_9bbe_f1cab97feaa5.slice/crio-55661385adb84c291a4b82b14d0b1df7087f1a920f2b74119f5d4a753d008eca WatchSource:0}: Error finding container 55661385adb84c291a4b82b14d0b1df7087f1a920f2b74119f5d4a753d008eca: Status 404 returned error can't find the container with id 55661385adb84c291a4b82b14d0b1df7087f1a920f2b74119f5d4a753d008eca
Jul 28 12:35:33.022773 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:33.022743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"
Jul 28 12:35:33.187123 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:33.187055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c"]
Jul 28 12:35:33.190966 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:33.190933    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddedb54_cc14_4cb7_ac99_4683df900722.slice/crio-2fce4a2fd1ee3d6dbfa492387940784b47b2d7e3c8c37425b07f1192f2aa3a27 WatchSource:0}: Error finding container 2fce4a2fd1ee3d6dbfa492387940784b47b2d7e3c8c37425b07f1192f2aa3a27: Status 404 returned error can't find the container with id 2fce4a2fd1ee3d6dbfa492387940784b47b2d7e3c8c37425b07f1192f2aa3a27
Jul 28 12:35:33.313528 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:33.313490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw" event={"ID":"830400de-c238-479c-9bbe-f1cab97feaa5","Type":"ContainerStarted","Data":"55661385adb84c291a4b82b14d0b1df7087f1a920f2b74119f5d4a753d008eca"}
Jul 28 12:35:33.314704 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:33.314678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c" event={"ID":"eddedb54-cc14-4cb7-ac99-4683df900722","Type":"ContainerStarted","Data":"2fce4a2fd1ee3d6dbfa492387940784b47b2d7e3c8c37425b07f1192f2aa3a27"}
Jul 28 12:35:35.004244 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:35.004195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:35.004659 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:35.004329    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:35.004659 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:35.004351    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-964f6bf67-wct99: secret "image-registry-tls" not found
Jul 28 12:35:35.004659 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:35.004401    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls podName:56de39ce-f39c-4563-9746-e630fb800d6a nodeName:}" failed. No retries permitted until 2026-07-28 12:35:39.004386826 +0000 UTC m=+45.490011398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls") pod "image-registry-964f6bf67-wct99" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a") : secret "image-registry-tls" not found
Jul 28 12:35:35.323992 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:35.323901    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn" event={"ID":"d6979445-3b85-4f61-a570-b21e8b788ffd","Type":"ContainerStarted","Data":"6e1461c74c81801206e62976a66cc7563489a13207d822a42edae8de3d7ab540"}
Jul 28 12:35:35.324143 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:35.324013    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn" event={"ID":"d6979445-3b85-4f61-a570-b21e8b788ffd","Type":"ContainerStarted","Data":"b25545128b41bbb1f554506412eb170988d5181f2110de3e2163f2ac8d123f14"}
Jul 28 12:35:35.343407 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:35.343343    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-bt9cn" podStartSLOduration=2.795910245 podStartE2EDuration="5.343321897s" totalImagesPullingTime="2.547411652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:35.341670915 +0000 UTC m=+41.827295506" watchObservedRunningTime="2026-07-28 12:35:35.343321897 +0000 UTC m=+41.828946491"
Jul 28 12:35:36.328683 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.328641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw" event={"ID":"830400de-c238-479c-9bbe-f1cab97feaa5","Type":"ContainerStarted","Data":"8778b2c89c81f59415f1866fb0c7fefd4cf5a3e1f8f5c26ae85cabfd2cad36f6"}
Jul 28 12:35:36.346147 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.346091    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-vxhgw" podStartSLOduration=1.889583294 podStartE2EDuration="4.346070873s" totalImagesPullingTime="2.456487579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:36.345603016 +0000 UTC m=+42.831227609" watchObservedRunningTime="2026-07-28 12:35:36.346070873 +0000 UTC m=+42.831695465"
Jul 28 12:35:36.717789 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.717754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:36.717965 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.717818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:36.717965 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.717870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:36.717965 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:36.717916    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:36.718140 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:36.717976    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls podName:1f8d8022-51f4-425c-bc23-3023a521481f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:44.717958975 +0000 UTC m=+51.203583556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls") pod "dns-default-q4c6v" (UID: "1f8d8022-51f4-425c-bc23-3023a521481f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:36.718140 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:36.718028    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:36.718140 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:36.718108    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls podName:90d16b4e-aa12-40c5-a466-2202e6a14ff3 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:44.718089213 +0000 UTC m=+51.203713792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfppf" (UID: "90d16b4e-aa12-40c5-a466-2202e6a14ff3") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:36.718416 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:36.718393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90d16b4e-aa12-40c5-a466-2202e6a14ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:37.334255 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:37.334155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c" event={"ID":"eddedb54-cc14-4cb7-ac99-4683df900722","Type":"ContainerStarted","Data":"1a1ff2e729a37bb09b27e5f45b301d51bfb6cce9747ddc39c7c04238bdd48523"}
Jul 28 12:35:37.353215 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:37.353160    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dq99c" podStartSLOduration=1.507080394 podStartE2EDuration="5.353142304s" totalImagesPullingTime="3.84606191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:37.35210151 +0000 UTC m=+43.837726101" watchObservedRunningTime="2026-07-28 12:35:37.353142304 +0000 UTC m=+43.838766896"
Jul 28 12:35:39.037298 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:39.037257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:39.037772 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:39.037376    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:39.037772 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:39.037388    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-964f6bf67-wct99: secret "image-registry-tls" not found
Jul 28 12:35:39.037772 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:35:39.037435    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls podName:56de39ce-f39c-4563-9746-e630fb800d6a nodeName:}" failed. No retries permitted until 2026-07-28 12:35:47.037421551 +0000 UTC m=+53.523046121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls") pod "image-registry-964f6bf67-wct99" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a") : secret "image-registry-tls" not found
Jul 28 12:35:44.783439 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.783397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:44.783852 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.783506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:44.785996 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.785962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f8d8022-51f4-425c-bc23-3023a521481f-metrics-tls\") pod \"dns-default-q4c6v\" (UID: \"1f8d8022-51f4-425c-bc23-3023a521481f\") " pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:44.786118 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.785964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90d16b4e-aa12-40c5-a466-2202e6a14ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfppf\" (UID: \"90d16b4e-aa12-40c5-a466-2202e6a14ff3\") " pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:44.825082 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.825055    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfppf"
Jul 28 12:35:44.842386 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.842356    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:44.980350 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:44.980314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfppf"]
Jul 28 12:35:44.983141 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:44.983103    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d16b4e_aa12_40c5_a466_2202e6a14ff3.slice/crio-d8593d34fddcd7289ec8d130a65e3d3c2747e8197782d547b0fbe310ac67c1f8 WatchSource:0}: Error finding container d8593d34fddcd7289ec8d130a65e3d3c2747e8197782d547b0fbe310ac67c1f8: Status 404 returned error can't find the container with id d8593d34fddcd7289ec8d130a65e3d3c2747e8197782d547b0fbe310ac67c1f8
Jul 28 12:35:45.007956 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:45.007929    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q4c6v"]
Jul 28 12:35:45.012473 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:45.012451    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8d8022_51f4_425c_bc23_3023a521481f.slice/crio-0c310cfbe5ff73ae67c4cfe81b53b9f6da4aae99cf0df62d4e691e75ef9c3025 WatchSource:0}: Error finding container 0c310cfbe5ff73ae67c4cfe81b53b9f6da4aae99cf0df62d4e691e75ef9c3025: Status 404 returned error can't find the container with id 0c310cfbe5ff73ae67c4cfe81b53b9f6da4aae99cf0df62d4e691e75ef9c3025
Jul 28 12:35:45.360070 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:45.360027    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfppf" event={"ID":"90d16b4e-aa12-40c5-a466-2202e6a14ff3","Type":"ContainerStarted","Data":"d8593d34fddcd7289ec8d130a65e3d3c2747e8197782d547b0fbe310ac67c1f8"}
Jul 28 12:35:45.361022 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:45.360996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4c6v" event={"ID":"1f8d8022-51f4-425c-bc23-3023a521481f","Type":"ContainerStarted","Data":"0c310cfbe5ff73ae67c4cfe81b53b9f6da4aae99cf0df62d4e691e75ef9c3025"}
Jul 28 12:35:47.106658 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.106343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:47.109919 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.109887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"image-registry-964f6bf67-wct99\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") " pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:47.140829 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.140803    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:47.308133 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.308101    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:35:47.312322 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:47.312237    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56de39ce_f39c_4563_9746_e630fb800d6a.slice/crio-fff0584f1e8702df5be24719bb90ab369ce18e9c17738e6a3119235fdfb87c48 WatchSource:0}: Error finding container fff0584f1e8702df5be24719bb90ab369ce18e9c17738e6a3119235fdfb87c48: Status 404 returned error can't find the container with id fff0584f1e8702df5be24719bb90ab369ce18e9c17738e6a3119235fdfb87c48
Jul 28 12:35:47.372681 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.372632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4c6v" event={"ID":"1f8d8022-51f4-425c-bc23-3023a521481f","Type":"ContainerStarted","Data":"34b12cf4bd59cd314389d6ad1170139798da8e9997ca896928be8f7bb3c3da91"}
Jul 28 12:35:47.372826 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.372692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4c6v" event={"ID":"1f8d8022-51f4-425c-bc23-3023a521481f","Type":"ContainerStarted","Data":"7854046880bc745647c97d9d19050447ff3e84d4edf23bf0ae5b5fb70efadd55"}
Jul 28 12:35:47.372913 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.372893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:47.374656 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.374625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964f6bf67-wct99" event={"ID":"56de39ce-f39c-4563-9746-e630fb800d6a","Type":"ContainerStarted","Data":"fff0584f1e8702df5be24719bb90ab369ce18e9c17738e6a3119235fdfb87c48"}
Jul 28 12:35:47.408068 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:47.407996    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q4c6v" podStartSLOduration=17.335672564 podStartE2EDuration="19.407973523s" totalImagesPullingTime="2.072300959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:47.407471181 +0000 UTC m=+53.893095774" watchObservedRunningTime="2026-07-28 12:35:47.407973523 +0000 UTC m=+53.893598118"
Jul 28 12:35:48.380550 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:48.380432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964f6bf67-wct99" event={"ID":"56de39ce-f39c-4563-9746-e630fb800d6a","Type":"ContainerStarted","Data":"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"}
Jul 28 12:35:48.380975 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:48.380758    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:35:48.405204 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:48.404866    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-964f6bf67-wct99" podStartSLOduration=17.404844713 podStartE2EDuration="17.404844713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:48.403484516 +0000 UTC m=+54.889109108" watchObservedRunningTime="2026-07-28 12:35:48.404844713 +0000 UTC m=+54.890469285"
Jul 28 12:35:51.392482 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:51.392449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfppf" event={"ID":"90d16b4e-aa12-40c5-a466-2202e6a14ff3","Type":"ContainerStarted","Data":"8d6cfc15f73197dd81500aa8db17355188c3f3c9551b061ab33133974d820664"}
Jul 28 12:35:55.333052 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.333023    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k48sz"
Jul 28 12:35:55.408379 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.408345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfppf" event={"ID":"90d16b4e-aa12-40c5-a466-2202e6a14ff3","Type":"ContainerStarted","Data":"8668627f66aaa279ff079df2c60b996c2619e82b9f473f0ec316d5801bff4784"}
Jul 28 12:35:55.806072 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.806042    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"]
Jul 28 12:35:55.808514 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.808492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.812388 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.812362    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 12:35:55.812566 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.812406    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 12:35:55.812566 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.812534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-chpgj\""
Jul 28 12:35:55.820602 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.820558    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:35:55.822393 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.822372    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"]
Jul 28 12:35:55.871613 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.871584    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585cb7d774-wgzt6"]
Jul 28 12:35:55.873999 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.873970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.878210 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.878182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.878354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.878259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.901760 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.901707    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585cb7d774-wgzt6"]
Jul 28 12:35:55.902780 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.902758    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-dn64j"]
Jul 28 12:35:55.905126 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.905107    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:35:55.907991 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.907968    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-967tm\""
Jul 28 12:35:55.908125 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.908055    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:35:55.908812 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.908792    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:35:55.923832 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.923715    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dn64j"]
Jul 28 12:35:55.979301 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb90edc5-d7a7-4915-9e67-5246cdc52234-ca-trust-extracted\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979301 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdn4\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-kube-api-access-kkdn4\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979513 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-image-registry-private-configuration\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979513 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsf5\" (UniqueName: \"kubernetes.io/projected/68125006-528f-4eb7-b4e1-9a29c9443b52-kube-api-access-djsf5\") pod \"downloads-7c97dc4555-dn64j\" (UID: \"68125006-528f-4eb7-b4e1-9a29c9443b52\") " pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:35:55.979596 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-certificates\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979646 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.979685 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-bound-sa-token\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979756 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-tls\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979756 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-trusted-ca\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979834 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-installation-pull-secrets\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:55.979834 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.979815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.980377 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.980351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:55.982694 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:55.982670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05b5ffc3-65b4-47c5-9e97-06f9cf9402db-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6lsv8\" (UID: \"05b5ffc3-65b4-47c5-9e97-06f9cf9402db\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:56.081308 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb90edc5-d7a7-4915-9e67-5246cdc52234-ca-trust-extracted\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081308 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdn4\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-kube-api-access-kkdn4\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081308 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-image-registry-private-configuration\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081308 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsf5\" (UniqueName: \"kubernetes.io/projected/68125006-528f-4eb7-b4e1-9a29c9443b52-kube-api-access-djsf5\") pod \"downloads-7c97dc4555-dn64j\" (UID: \"68125006-528f-4eb7-b4e1-9a29c9443b52\") " pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:35:56.081710 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-certificates\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081710 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-bound-sa-token\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081710 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-tls\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081710 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-trusted-ca\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081710 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-installation-pull-secrets\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.081992 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.081708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb90edc5-d7a7-4915-9e67-5246cdc52234-ca-trust-extracted\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.082338 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.082306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-certificates\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.082769 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.082741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb90edc5-d7a7-4915-9e67-5246cdc52234-trusted-ca\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.084371 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.084349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-installation-pull-secrets\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.084457 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.084412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb90edc5-d7a7-4915-9e67-5246cdc52234-image-registry-private-configuration\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.084838 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.084819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-registry-tls\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.102408 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.102358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdn4\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-kube-api-access-kkdn4\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.102856 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.102838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsf5\" (UniqueName: \"kubernetes.io/projected/68125006-528f-4eb7-b4e1-9a29c9443b52-kube-api-access-djsf5\") pod \"downloads-7c97dc4555-dn64j\" (UID: \"68125006-528f-4eb7-b4e1-9a29c9443b52\") " pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:35:56.103080 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.103061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb90edc5-d7a7-4915-9e67-5246cdc52234-bound-sa-token\") pod \"image-registry-585cb7d774-wgzt6\" (UID: \"cb90edc5-d7a7-4915-9e67-5246cdc52234\") " pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.118844 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.118802    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"
Jul 28 12:35:56.185384 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.185095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:56.215388 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.215110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:35:56.289016 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.288616    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8"]
Jul 28 12:35:56.293896 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:56.293862    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b5ffc3_65b4_47c5_9e97_06f9cf9402db.slice/crio-0e177bd8607b903df7b62875c887874ccc7c21656da99a964e439990bbe42616 WatchSource:0}: Error finding container 0e177bd8607b903df7b62875c887874ccc7c21656da99a964e439990bbe42616: Status 404 returned error can't find the container with id 0e177bd8607b903df7b62875c887874ccc7c21656da99a964e439990bbe42616
Jul 28 12:35:56.350251 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.350225    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585cb7d774-wgzt6"]
Jul 28 12:35:56.352943 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:56.352918    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb90edc5_d7a7_4915_9e67_5246cdc52234.slice/crio-7ac126729fea3ec267daebe6d4653273b4ec3cb5c92a8aaf0485e5bb4c6308fc WatchSource:0}: Error finding container 7ac126729fea3ec267daebe6d4653273b4ec3cb5c92a8aaf0485e5bb4c6308fc: Status 404 returned error can't find the container with id 7ac126729fea3ec267daebe6d4653273b4ec3cb5c92a8aaf0485e5bb4c6308fc
Jul 28 12:35:56.380044 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.380015    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dn64j"]
Jul 28 12:35:56.383598 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:35:56.383568    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68125006_528f_4eb7_b4e1_9a29c9443b52.slice/crio-f7ed5c56d67fb929e517718dc67a17a5f5ca33776f709348f5f4965c0237dd7e WatchSource:0}: Error finding container f7ed5c56d67fb929e517718dc67a17a5f5ca33776f709348f5f4965c0237dd7e: Status 404 returned error can't find the container with id f7ed5c56d67fb929e517718dc67a17a5f5ca33776f709348f5f4965c0237dd7e
Jul 28 12:35:56.413615 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.413582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6" event={"ID":"cb90edc5-d7a7-4915-9e67-5246cdc52234","Type":"ContainerStarted","Data":"7ac126729fea3ec267daebe6d4653273b4ec3cb5c92a8aaf0485e5bb4c6308fc"}
Jul 28 12:35:56.414996 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.414966    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8" event={"ID":"05b5ffc3-65b4-47c5-9e97-06f9cf9402db","Type":"ContainerStarted","Data":"0e177bd8607b903df7b62875c887874ccc7c21656da99a964e439990bbe42616"}
Jul 28 12:35:56.416353 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:56.416328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dn64j" event={"ID":"68125006-528f-4eb7-b4e1-9a29c9443b52","Type":"ContainerStarted","Data":"f7ed5c56d67fb929e517718dc67a17a5f5ca33776f709348f5f4965c0237dd7e"}
Jul 28 12:35:57.383011 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:57.382933    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q4c6v"
Jul 28 12:35:57.423089 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:57.423056    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6" event={"ID":"cb90edc5-d7a7-4915-9e67-5246cdc52234","Type":"ContainerStarted","Data":"3dad95bb38d1fda0c6c224855197ef662d6465f68a897a0c6da7463ced488be2"}
Jul 28 12:35:57.423692 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:57.423667    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:35:57.450738 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:57.450665    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6" podStartSLOduration=2.45064457 podStartE2EDuration="2.45064457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:57.449251113 +0000 UTC m=+63.934875706" watchObservedRunningTime="2026-07-28 12:35:57.45064457 +0000 UTC m=+63.936269161"
Jul 28 12:35:59.432058 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.432022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfppf" event={"ID":"90d16b4e-aa12-40c5-a466-2202e6a14ff3","Type":"ContainerStarted","Data":"251169dc31528c29033d689906875d7efd46a0731715d6ed2df3ef68a5831e71"}
Jul 28 12:35:59.457233 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.457180    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lfppf" podStartSLOduration=18.063468582 podStartE2EDuration="31.457160326s" totalImagesPullingTime="13.393691744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:59.455154071 +0000 UTC m=+65.940778665" watchObservedRunningTime="2026-07-28 12:35:59.457160326 +0000 UTC m=+65.942784919"
Jul 28 12:35:59.822793 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.822691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:59.822793 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.822782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:59.825683 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.825655    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:35:59.825683 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.825682    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:35:59.835567 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.835544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911090a5-d1b2-49eb-ab42-3a239ccaa4f0-metrics-certs\") pod \"network-metrics-daemon-qhblp\" (UID: \"911090a5-d1b2-49eb-ab42-3a239ccaa4f0\") " pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:35:59.835663 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.835626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdff30c9-8adc-4b6b-8e8c-a771af042897-original-pull-secret\") pod \"global-pull-secret-syncer-8fcfw\" (UID: \"cdff30c9-8adc-4b6b-8e8c-a771af042897\") " pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:35:59.923466 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.923430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:59.926402 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.926373    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp69p\" (UniqueName: \"kubernetes.io/projected/611e17b6-2857-4c4d-93e1-77d997af210d-kube-api-access-rp69p\") pod \"network-check-target-5cmjc\" (UID: \"611e17b6-2857-4c4d-93e1-77d997af210d\") " pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:35:59.930128 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:35:59.930104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fcfw"
Jul 28 12:36:00.025739 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.025693    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf2p5\""
Jul 28 12:36:00.034284 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.033984    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhblp"
Jul 28 12:36:00.111467 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.111384    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fcfw"]
Jul 28 12:36:00.114945 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:36:00.114878    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdff30c9_8adc_4b6b_8e8c_a771af042897.slice/crio-b84ff8fad5bcf987c7330c21612662151c96b7a92e4d341077f88084753e8f35 WatchSource:0}: Error finding container b84ff8fad5bcf987c7330c21612662151c96b7a92e4d341077f88084753e8f35: Status 404 returned error can't find the container with id b84ff8fad5bcf987c7330c21612662151c96b7a92e4d341077f88084753e8f35
Jul 28 12:36:00.202749 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.202706    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhblp"]
Jul 28 12:36:00.204993 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:36:00.204965    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911090a5_d1b2_49eb_ab42_3a239ccaa4f0.slice/crio-f0737884c2d904fc1fb952e122c964c10ec70d364869c395fc55013845df4f49 WatchSource:0}: Error finding container f0737884c2d904fc1fb952e122c964c10ec70d364869c395fc55013845df4f49: Status 404 returned error can't find the container with id f0737884c2d904fc1fb952e122c964c10ec70d364869c395fc55013845df4f49
Jul 28 12:36:00.226512 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.226488    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc56m\""
Jul 28 12:36:00.234015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.233988    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:36:00.264217 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.264185    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"]
Jul 28 12:36:00.285430 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.285395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"]
Jul 28 12:36:00.285576 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.285524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.289340 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.289312    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 12:36:00.290489 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.290466    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 12:36:00.290635 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.290475    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2hzb6\""
Jul 28 12:36:00.290700 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.290638    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:36:00.291012 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.290996    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:36:00.291187 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.291174    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:36:00.327154 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.327121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.327288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.327210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.327288 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.327248    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6td4\" (UniqueName: \"kubernetes.io/projected/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-kube-api-access-k6td4\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.327358 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.327283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.427681 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.427655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.427895 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.427738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.427895 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.427784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6td4\" (UniqueName: \"kubernetes.io/projected/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-kube-api-access-k6td4\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.427895 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.427815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.427895 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:00.427846    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 12:36:00.428156 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:00.427919    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls podName:574f5331-12b8-40bc-ab0d-93ce4f00f1a4 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.927899127 +0000 UTC m=+67.413523699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-7c8gd" (UID: "574f5331-12b8-40bc-ab0d-93ce4f00f1a4") : secret "prometheus-operator-tls" not found
Jul 28 12:36:00.428508 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.428485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.428839 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.428791    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cmjc"]
Jul 28 12:36:00.430740 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.430698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.432253 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:36:00.432228    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611e17b6_2857_4c4d_93e1_77d997af210d.slice/crio-3037486a863b12722901fcc30b09ea9de3da5a64a503747eb87240d3fc90e67f WatchSource:0}: Error finding container 3037486a863b12722901fcc30b09ea9de3da5a64a503747eb87240d3fc90e67f: Status 404 returned error can't find the container with id 3037486a863b12722901fcc30b09ea9de3da5a64a503747eb87240d3fc90e67f
Jul 28 12:36:00.437261 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.437239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhblp" event={"ID":"911090a5-d1b2-49eb-ab42-3a239ccaa4f0","Type":"ContainerStarted","Data":"f0737884c2d904fc1fb952e122c964c10ec70d364869c395fc55013845df4f49"}
Jul 28 12:36:00.438450 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.438425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6td4\" (UniqueName: \"kubernetes.io/projected/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-kube-api-access-k6td4\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.438925 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.438898    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8" event={"ID":"05b5ffc3-65b4-47c5-9e97-06f9cf9402db","Type":"ContainerStarted","Data":"68f08d3a6860bb2bc4a1547dbd1e01cb15ea34aa2c532476ab37074aa542573a"}
Jul 28 12:36:00.440114 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.440091    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fcfw" event={"ID":"cdff30c9-8adc-4b6b-8e8c-a771af042897","Type":"ContainerStarted","Data":"b84ff8fad5bcf987c7330c21612662151c96b7a92e4d341077f88084753e8f35"}
Jul 28 12:36:00.481154 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.481108    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6lsv8" podStartSLOduration=2.390735271 podStartE2EDuration="5.481091436s" totalImagesPullingTime="3.090356165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:00.481017336 +0000 UTC m=+66.966641932" watchObservedRunningTime="2026-07-28 12:36:00.481091436 +0000 UTC m=+66.966716027"
Jul 28 12:36:00.931896 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.931854    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:00.940617 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:00.940559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/574f5331-12b8-40bc-ab0d-93ce4f00f1a4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7c8gd\" (UID: \"574f5331-12b8-40bc-ab0d-93ce4f00f1a4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:01.198006 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:01.197927    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"
Jul 28 12:36:01.427101 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:01.426379    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7c8gd"]
Jul 28 12:36:01.449076 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:01.448925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cmjc" event={"ID":"611e17b6-2857-4c4d-93e1-77d997af210d","Type":"ContainerStarted","Data":"dfd708767785578289562bfb2da313126da31435e536262441bd14dd7cbd8779"}
Jul 28 12:36:01.449076 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:01.448978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cmjc" event={"ID":"611e17b6-2857-4c4d-93e1-77d997af210d","Type":"ContainerStarted","Data":"3037486a863b12722901fcc30b09ea9de3da5a64a503747eb87240d3fc90e67f"}
Jul 28 12:36:01.449076 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:01.449002    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:36:02.456003 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:02.455963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhblp" event={"ID":"911090a5-d1b2-49eb-ab42-3a239ccaa4f0","Type":"ContainerStarted","Data":"139cd268787d28d94de740138a53803c13f0b728e7dfc20d2c117863da471add"}
Jul 28 12:36:02.456438 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:02.456025    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhblp" event={"ID":"911090a5-d1b2-49eb-ab42-3a239ccaa4f0","Type":"ContainerStarted","Data":"8399dad6292eeaeee5296f00e46cb5d0890c00118a6276d25e6383320edc5e28"}
Jul 28 12:36:02.458736 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:02.458697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd" event={"ID":"574f5331-12b8-40bc-ab0d-93ce4f00f1a4","Type":"ContainerStarted","Data":"bd84e7794b96371d2996f3a2e3bfcb82334875978cf4fc47e1cfd27b2a42208f"}
Jul 28 12:36:02.477840 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:02.477792    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qhblp" podStartSLOduration=66.692758223 podStartE2EDuration="1m8.477776631s" totalImagesPullingTime="1.785018408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:02.476599284 +0000 UTC m=+68.962223876" watchObservedRunningTime="2026-07-28 12:36:02.477776631 +0000 UTC m=+68.963401223"
Jul 28 12:36:02.477995 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:02.477871    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5cmjc" podStartSLOduration=68.477867204 podStartE2EDuration="1m8.477867204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:01.473198499 +0000 UTC m=+67.958823092" watchObservedRunningTime="2026-07-28 12:36:02.477867204 +0000 UTC m=+68.963491795"
Jul 28 12:36:05.474661 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.474621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd" event={"ID":"574f5331-12b8-40bc-ab0d-93ce4f00f1a4","Type":"ContainerStarted","Data":"c44e85f7f0b93420589a04aa81dff4209ed8072aa3daf9c9b5a41ed260a3f1ce"}
Jul 28 12:36:05.475212 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.474735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd" event={"ID":"574f5331-12b8-40bc-ab0d-93ce4f00f1a4","Type":"ContainerStarted","Data":"c4249ce3a13e29e7befab54da7311b2cca5c6563ee935787ec87cb7483de32f7"}
Jul 28 12:36:05.476851 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.476822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fcfw" event={"ID":"cdff30c9-8adc-4b6b-8e8c-a771af042897","Type":"ContainerStarted","Data":"445794ea404612e1381e039693d51986eeae09cc82b20470f91cbd85f5a33c70"}
Jul 28 12:36:05.496763 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.496703    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-7c8gd" podStartSLOduration=2.085383604 podStartE2EDuration="5.496685907s" totalImagesPullingTime="3.411302303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:05.494809995 +0000 UTC m=+71.980434589" watchObservedRunningTime="2026-07-28 12:36:05.496685907 +0000 UTC m=+71.982310500"
Jul 28 12:36:05.516646 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.516584    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8fcfw" podStartSLOduration=66.562027666 podStartE2EDuration="1m11.516567756s" totalImagesPullingTime="4.95454009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:05.516030092 +0000 UTC m=+72.001654695" watchObservedRunningTime="2026-07-28 12:36:05.516567756 +0000 UTC m=+72.002192354"
Jul 28 12:36:05.827349 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:05.827264    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:36:07.740615 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.739438    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dsl6j"]
Jul 28 12:36:07.742595 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.742085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.746637 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.746461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:36:07.746637 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.746490    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g65q6\""
Jul 28 12:36:07.746637 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.746539    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:36:07.747241 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.747084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.795851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-root\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.795904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.795931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.795956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t22v\" (UniqueName: \"kubernetes.io/projected/45919c63-28b1-46cc-9a18-f774140260ba-kube-api-access-6t22v\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.795984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-sys\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.796007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-wtmp\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.796047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-metrics-client-ca\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.796093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.796737 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.796113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-textfile\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899144 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-root\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899153    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t22v\" (UniqueName: \"kubernetes.io/projected/45919c63-28b1-46cc-9a18-f774140260ba-kube-api-access-6t22v\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-sys\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-wtmp\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899324 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-metrics-client-ca\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899600 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899600 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-textfile\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899804 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-textfile\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.899880 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-sys\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.900403 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.899984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-wtmp\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.900403 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:07.900065    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:36:07.900403 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:07.900141    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls podName:45919c63-28b1-46cc-9a18-f774140260ba nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.400120424 +0000 UTC m=+74.885745008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls") pod "node-exporter-dsl6j" (UID: "45919c63-28b1-46cc-9a18-f774140260ba") : secret "node-exporter-tls" not found
Jul 28 12:36:07.900403 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.900189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45919c63-28b1-46cc-9a18-f774140260ba-root\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.900690 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.900466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-metrics-client-ca\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.900948 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.900923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.903322 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.903276    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:07.913690 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:07.913665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t22v\" (UniqueName: \"kubernetes.io/projected/45919c63-28b1-46cc-9a18-f774140260ba-kube-api-access-6t22v\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:08.404260 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:08.404222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:08.404443 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:08.404415    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:36:08.404509 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:08.404496    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls podName:45919c63-28b1-46cc-9a18-f774140260ba nodeName:}" failed. No retries permitted until 2026-07-28 12:36:09.404477034 +0000 UTC m=+75.890101609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls") pod "node-exporter-dsl6j" (UID: "45919c63-28b1-46cc-9a18-f774140260ba") : secret "node-exporter-tls" not found
Jul 28 12:36:09.414611 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:09.414576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:09.417102 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:09.417078    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45919c63-28b1-46cc-9a18-f774140260ba-node-exporter-tls\") pod \"node-exporter-dsl6j\" (UID: \"45919c63-28b1-46cc-9a18-f774140260ba\") " pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:09.554039 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:09.554003    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsl6j"
Jul 28 12:36:15.489201 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:36:15.489168    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45919c63_28b1_46cc_9a18_f774140260ba.slice/crio-cefcc9862944c226928775109c8fc5ed5c3ca54da27dad2bcffebaee01adb022 WatchSource:0}: Error finding container cefcc9862944c226928775109c8fc5ed5c3ca54da27dad2bcffebaee01adb022: Status 404 returned error can't find the container with id cefcc9862944c226928775109c8fc5ed5c3ca54da27dad2bcffebaee01adb022
Jul 28 12:36:15.518344 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:15.518310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsl6j" event={"ID":"45919c63-28b1-46cc-9a18-f774140260ba","Type":"ContainerStarted","Data":"cefcc9862944c226928775109c8fc5ed5c3ca54da27dad2bcffebaee01adb022"}
Jul 28 12:36:16.524808 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:16.524763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dn64j" event={"ID":"68125006-528f-4eb7-b4e1-9a29c9443b52","Type":"ContainerStarted","Data":"802972c357202fa41a17e348f4a43a87b48252cd9f6f8f668f164ba2f78b654e"}
Jul 28 12:36:16.525311 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:16.524976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:36:16.540044 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:16.540010    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-dn64j"
Jul 28 12:36:16.547963 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:16.547913    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-dn64j" podStartSLOduration=2.151702999 podStartE2EDuration="21.547893177s" totalImagesPullingTime="19.396190178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:16.545253879 +0000 UTC m=+83.030878471" watchObservedRunningTime="2026-07-28 12:36:16.547893177 +0000 UTC m=+83.033517950"
Jul 28 12:36:17.530868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:17.530779    2558 generic.go:356] "Generic (PLEG): container finished" podID="45919c63-28b1-46cc-9a18-f774140260ba" containerID="dcd4053835c207a9b1ab8ec4cadb5ecbf8aea6ba3c6148d2ce40c84c9dae90ba" exitCode=0
Jul 28 12:36:17.530868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:17.530858    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsl6j" event={"ID":"45919c63-28b1-46cc-9a18-f774140260ba","Type":"ContainerDied","Data":"dcd4053835c207a9b1ab8ec4cadb5ecbf8aea6ba3c6148d2ce40c84c9dae90ba"}
Jul 28 12:36:18.537433 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:18.537387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsl6j" event={"ID":"45919c63-28b1-46cc-9a18-f774140260ba","Type":"ContainerStarted","Data":"f9cbab548b7bdac7066dbb2b2dda22a571767fe99addc5371de228a6246b9ea1"}
Jul 28 12:36:18.537932 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:18.537450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsl6j" event={"ID":"45919c63-28b1-46cc-9a18-f774140260ba","Type":"ContainerStarted","Data":"98bae3eee9472d83e7d34ab1a6310711ad74487ca07ed866a5dfb3b9765e61f5"}
Jul 28 12:36:18.561494 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:18.561432    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dsl6j" podStartSLOduration=10.071491967 podStartE2EDuration="11.561412367s" totalImagesPullingTime="1.36823592s" totalInitContainerRuntime="121.68448ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:18.560141435 +0000 UTC m=+85.045766027" watchObservedRunningTime="2026-07-28 12:36:18.561412367 +0000 UTC m=+85.047036960"
Jul 28 12:36:19.437452 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:19.437421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-585cb7d774-wgzt6"
Jul 28 12:36:20.844002 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:20.843942    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-964f6bf67-wct99" podUID="56de39ce-f39c-4563-9746-e630fb800d6a" containerName="registry" containerID="cri-o://674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d" gracePeriod=30
Jul 28 12:36:21.156529 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.156494    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:36:21.330790 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330746    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330803    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330842    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330869    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330913    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330947    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.330985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.330978    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.331300 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.331022    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl5q4\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4\") pod \"56de39ce-f39c-4563-9746-e630fb800d6a\" (UID: \"56de39ce-f39c-4563-9746-e630fb800d6a\") "
Jul 28 12:36:21.331439 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.331401    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:21.331944 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.331911    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:21.335559 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.335527    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4" (OuterVolumeSpecName: "kube-api-access-bl5q4") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "kube-api-access-bl5q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:21.338913 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.338809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:21.338913 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.338826    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:21.339038 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.338954    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:21.339038 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.338991    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:21.341436 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.341410    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "56de39ce-f39c-4563-9746-e630fb800d6a" (UID: "56de39ce-f39c-4563-9746-e630fb800d6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432557    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5q4\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-kube-api-access-bl5q4\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432602    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-image-registry-private-configuration\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432616    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de39ce-f39c-4563-9746-e630fb800d6a-ca-trust-extracted\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432628    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de39ce-f39c-4563-9746-e630fb800d6a-installation-pull-secrets\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432641    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-bound-sa-token\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.432648 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432654    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de39ce-f39c-4563-9746-e630fb800d6a-registry-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.433015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432664    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-trusted-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.433015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.432674    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de39ce-f39c-4563-9746-e630fb800d6a-registry-certificates\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:36:21.551027 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.550989    2558 generic.go:356] "Generic (PLEG): container finished" podID="56de39ce-f39c-4563-9746-e630fb800d6a" containerID="674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d" exitCode=0
Jul 28 12:36:21.551027 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.551042    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964f6bf67-wct99" event={"ID":"56de39ce-f39c-4563-9746-e630fb800d6a","Type":"ContainerDied","Data":"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"}
Jul 28 12:36:21.551263 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.551075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964f6bf67-wct99"
Jul 28 12:36:21.551263 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.551123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964f6bf67-wct99" event={"ID":"56de39ce-f39c-4563-9746-e630fb800d6a","Type":"ContainerDied","Data":"fff0584f1e8702df5be24719bb90ab369ce18e9c17738e6a3119235fdfb87c48"}
Jul 28 12:36:21.551263 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.551155    2558 scope.go:117] "RemoveContainer" containerID="674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"
Jul 28 12:36:21.560913 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.560889    2558 scope.go:117] "RemoveContainer" containerID="674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"
Jul 28 12:36:21.561210 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:36:21.561179    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d\": container with ID starting with 674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d not found: ID does not exist" containerID="674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"
Jul 28 12:36:21.561291 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.561222    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d"} err="failed to get container status \"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d\": rpc error: code = NotFound desc = could not find container \"674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d\": container with ID starting with 674efbfd54cd835818d4db2d9ce87face06496acd496090fc0f374ce9828172d not found: ID does not exist"
Jul 28 12:36:21.577670 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.577643    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:36:21.583208 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:21.583186    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-964f6bf67-wct99"]
Jul 28 12:36:22.017497 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:22.017468    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56de39ce-f39c-4563-9746-e630fb800d6a" path="/var/lib/kubelet/pods/56de39ce-f39c-4563-9746-e630fb800d6a/volumes"
Jul 28 12:36:32.461162 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:36:32.461128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5cmjc"
Jul 28 12:39:48.876422 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.876392    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"]
Jul 28 12:39:48.876868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.876668    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56de39ce-f39c-4563-9746-e630fb800d6a" containerName="registry"
Jul 28 12:39:48.876868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.876678    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="56de39ce-f39c-4563-9746-e630fb800d6a" containerName="registry"
Jul 28 12:39:48.876868 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.876733    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="56de39ce-f39c-4563-9746-e630fb800d6a" containerName="registry"
Jul 28 12:39:48.878457 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.878442    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:48.882663 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.882638    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:39:48.882825 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.882638    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 12:39:48.882825 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.882732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lpt25\""
Jul 28 12:39:48.882825 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.882644    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:39:48.882825 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.882805    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:39:48.891408 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.891386    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"]
Jul 28 12:39:48.909313 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.909284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5zr\" (UniqueName: \"kubernetes.io/projected/d7e6f06c-490d-4d16-be01-eecde79bc205-kube-api-access-7w5zr\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:48.909439 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:48.909322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6f06c-490d-4d16-be01-eecde79bc205-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.010266 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.010231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5zr\" (UniqueName: \"kubernetes.io/projected/d7e6f06c-490d-4d16-be01-eecde79bc205-kube-api-access-7w5zr\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.010431 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.010273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6f06c-490d-4d16-be01-eecde79bc205-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.012973 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.012949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6f06c-490d-4d16-be01-eecde79bc205-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.021259 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.021229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5zr\" (UniqueName: \"kubernetes.io/projected/d7e6f06c-490d-4d16-be01-eecde79bc205-kube-api-access-7w5zr\") pod \"managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6\" (UID: \"d7e6f06c-490d-4d16-be01-eecde79bc205\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.111503 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.111463    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"]
Jul 28 12:39:49.113672 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.113649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.116411 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.116391    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 12:39:49.133533 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.133471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"]
Jul 28 12:39:49.195074 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.195043    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"
Jul 28 12:39:49.211005 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.210977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fn75\" (UniqueName: \"kubernetes.io/projected/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-kube-api-access-4fn75\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.211123 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.211056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-tmp\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.211123 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.211100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-klusterlet-config\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.311562 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.311532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-tmp\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.311741 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.311579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-klusterlet-config\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.311741 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.311605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fn75\" (UniqueName: \"kubernetes.io/projected/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-kube-api-access-4fn75\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.312031 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.312005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-tmp\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.314222 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.314199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-klusterlet-config\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.320818 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.320796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fn75\" (UniqueName: \"kubernetes.io/projected/bfa09e2f-1b56-4e9b-8fda-e370ab70f29a-kube-api-access-4fn75\") pod \"klusterlet-addon-workmgr-db5dd9456-mmlh6\" (UID: \"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.329643 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.329622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6"]
Jul 28 12:39:49.331971 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:39:49.331947    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7e6f06c_490d_4d16_be01_eecde79bc205.slice/crio-822c96b1357198f80a304552d7d8177400ca2d200214f4332b016d5dab39c72f WatchSource:0}: Error finding container 822c96b1357198f80a304552d7d8177400ca2d200214f4332b016d5dab39c72f: Status 404 returned error can't find the container with id 822c96b1357198f80a304552d7d8177400ca2d200214f4332b016d5dab39c72f
Jul 28 12:39:49.423297 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.423212    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:49.549107 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:49.549083    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"]
Jul 28 12:39:49.551359 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:39:49.551317    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa09e2f_1b56_4e9b_8fda_e370ab70f29a.slice/crio-1d2ef08f41bfb6bace0c8716ff08e04fe6cc91fa5dfc05aef50e3aa058484900 WatchSource:0}: Error finding container 1d2ef08f41bfb6bace0c8716ff08e04fe6cc91fa5dfc05aef50e3aa058484900: Status 404 returned error can't find the container with id 1d2ef08f41bfb6bace0c8716ff08e04fe6cc91fa5dfc05aef50e3aa058484900
Jul 28 12:39:50.261686 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:50.261637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6" event={"ID":"d7e6f06c-490d-4d16-be01-eecde79bc205","Type":"ContainerStarted","Data":"822c96b1357198f80a304552d7d8177400ca2d200214f4332b016d5dab39c72f"}
Jul 28 12:39:50.263238 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:50.263207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6" event={"ID":"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a","Type":"ContainerStarted","Data":"1d2ef08f41bfb6bace0c8716ff08e04fe6cc91fa5dfc05aef50e3aa058484900"}
Jul 28 12:39:52.272173 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:52.272135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6" event={"ID":"d7e6f06c-490d-4d16-be01-eecde79bc205","Type":"ContainerStarted","Data":"019dfb6996c92ef6aec5fc84ef3fedf7db0c2b15b6eac1c0c4a62a6cc1b258a8"}
Jul 28 12:39:52.291176 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:52.291126    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fb48fdf8-n7dp6" podStartSLOduration=1.851001492 podStartE2EDuration="4.29111233s" totalImagesPullingTime="2.440110838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:52.29056723 +0000 UTC m=+298.776191821" watchObservedRunningTime="2026-07-28 12:39:52.29111233 +0000 UTC m=+298.776736921"
Jul 28 12:39:54.896352 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:54.896327    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:39:54.896352 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:54.896346    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:39:54.899046 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:54.899024    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:39:55.284687 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:55.284600    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6" event={"ID":"bfa09e2f-1b56-4e9b-8fda-e370ab70f29a","Type":"ContainerStarted","Data":"40a282f4a15c32eb159bdb7ce057e8021b549ddef661ce5c789fa59a3d07024f"}
Jul 28 12:39:55.284855 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:55.284790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:55.286492 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:55.286469    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6"
Jul 28 12:39:55.355355 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:39:55.355304    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-db5dd9456-mmlh6" podStartSLOduration=0.988811657 podStartE2EDuration="6.355284924s" totalImagesPullingTime="5.366473267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:55.319418591 +0000 UTC m=+301.805043182" watchObservedRunningTime="2026-07-28 12:39:55.355284924 +0000 UTC m=+301.840909509"
Jul 28 12:40:39.220856 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.220815    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"]
Jul 28 12:40:39.223205 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.223182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.226320 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.226296    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 12:40:39.226320 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.226315    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:40:39.226484 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.226320    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8fccz\""
Jul 28 12:40:39.226484 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.226374    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:40:39.240421 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.240396    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"]
Jul 28 12:40:39.242279 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.242257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchz7\" (UniqueName: \"kubernetes.io/projected/af5e018e-d289-4a2a-a555-f56afa392105-kube-api-access-gchz7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.242395 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.242305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/af5e018e-d289-4a2a-a555-f56afa392105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.343385 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.343349    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/af5e018e-d289-4a2a-a555-f56afa392105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.343554 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.343411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchz7\" (UniqueName: \"kubernetes.io/projected/af5e018e-d289-4a2a-a555-f56afa392105-kube-api-access-gchz7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.345888 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.345865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/af5e018e-d289-4a2a-a555-f56afa392105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.355213 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.355193    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchz7\" (UniqueName: \"kubernetes.io/projected/af5e018e-d289-4a2a-a555-f56afa392105-kube-api-access-gchz7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl\" (UID: \"af5e018e-d289-4a2a-a555-f56afa392105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.533345 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.533246    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:39.687100 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.687048    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"]
Jul 28 12:40:39.690218 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:40:39.690186    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e018e_d289_4a2a_a555_f56afa392105.slice/crio-8aa9363564be20f552216677a5d2ef711eb3c339ea0befa9748d9e9ce5d1ab6d WatchSource:0}: Error finding container 8aa9363564be20f552216677a5d2ef711eb3c339ea0befa9748d9e9ce5d1ab6d: Status 404 returned error can't find the container with id 8aa9363564be20f552216677a5d2ef711eb3c339ea0befa9748d9e9ce5d1ab6d
Jul 28 12:40:39.692086 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:39.692064    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:40:40.442917 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:40.442880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl" event={"ID":"af5e018e-d289-4a2a-a555-f56afa392105","Type":"ContainerStarted","Data":"8aa9363564be20f552216677a5d2ef711eb3c339ea0befa9748d9e9ce5d1ab6d"}
Jul 28 12:40:43.458260 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.458229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl" event={"ID":"af5e018e-d289-4a2a-a555-f56afa392105","Type":"ContainerStarted","Data":"e0ede0d0cf7eaf58d3bb994587734625821ad7a3da270f5bf3542274d42a219d"}
Jul 28 12:40:43.458669 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.458389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:40:43.483342 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.483176    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl" podStartSLOduration=1.262085142 podStartE2EDuration="4.483157012s" totalImagesPullingTime="3.22107187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:43.480913246 +0000 UTC m=+349.966537838" watchObservedRunningTime="2026-07-28 12:40:43.483157012 +0000 UTC m=+349.968781604"
Jul 28 12:40:43.617398 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.617357    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-2kxls"]
Jul 28 12:40:43.619762 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.619745    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.622604 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.622576    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:40:43.622751 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.622606    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 12:40:43.622890 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.622872    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-j8mcg\""
Jul 28 12:40:43.631949 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.631926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-2kxls"]
Jul 28 12:40:43.681226 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.681194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.681226 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.681230    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-cabundle0\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.681459 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.681271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lrv\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-kube-api-access-w6lrv\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.782096 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.782006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.782096 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.782044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-cabundle0\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.782313 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.782093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lrv\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-kube-api-access-w6lrv\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.782515 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:43.782495    2558 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 12:40:43.782657 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:43.782645    2558 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:40:43.782801 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:43.782784    2558 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:40:43.782935 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:43.782808    2558 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-2kxls: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:40:43.782935 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:43.782875    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates podName:7d55e463-14b0-4a7f-b7ba-82755cf6d53c nodeName:}" failed. No retries permitted until 2026-07-28 12:40:44.28285648 +0000 UTC m=+350.768481056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates") pod "keda-operator-77f648d694-2kxls" (UID: "7d55e463-14b0-4a7f-b7ba-82755cf6d53c") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:40:43.783153 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.783122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-cabundle0\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:43.791476 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:43.791424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lrv\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-kube-api-access-w6lrv\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:44.287030 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:44.286994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:44.287210 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:44.287118    2558 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:40:44.287210 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:44.287131    2558 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:40:44.287210 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:44.287139    2558 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-2kxls: references non-existent secret key: ca.crt
Jul 28 12:40:44.287210 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:40:44.287191    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates podName:7d55e463-14b0-4a7f-b7ba-82755cf6d53c nodeName:}" failed. No retries permitted until 2026-07-28 12:40:45.287178597 +0000 UTC m=+351.772803168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates") pod "keda-operator-77f648d694-2kxls" (UID: "7d55e463-14b0-4a7f-b7ba-82755cf6d53c") : references non-existent secret key: ca.crt
Jul 28 12:40:45.294875 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:45.294838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:45.297408 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:45.297387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d55e463-14b0-4a7f-b7ba-82755cf6d53c-certificates\") pod \"keda-operator-77f648d694-2kxls\" (UID: \"7d55e463-14b0-4a7f-b7ba-82755cf6d53c\") " pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:45.430287 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:45.430247    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:45.559064 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:45.559036    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-2kxls"]
Jul 28 12:40:45.560950 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:40:45.560921    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d55e463_14b0_4a7f_b7ba_82755cf6d53c.slice/crio-bb0586aebff298f22070d2e67e5dee17d628642e4b3314abf4b70768176fd27f WatchSource:0}: Error finding container bb0586aebff298f22070d2e67e5dee17d628642e4b3314abf4b70768176fd27f: Status 404 returned error can't find the container with id bb0586aebff298f22070d2e67e5dee17d628642e4b3314abf4b70768176fd27f
Jul 28 12:40:46.470811 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:46.470770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-2kxls" event={"ID":"7d55e463-14b0-4a7f-b7ba-82755cf6d53c","Type":"ContainerStarted","Data":"bb0586aebff298f22070d2e67e5dee17d628642e4b3314abf4b70768176fd27f"}
Jul 28 12:40:49.483972 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:49.483933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-2kxls" event={"ID":"7d55e463-14b0-4a7f-b7ba-82755cf6d53c","Type":"ContainerStarted","Data":"b94ddb382ccaf1cae5777268a2373138485ec1137320b74ec6c3ba75dff50992"}
Jul 28 12:40:49.484362 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:49.484082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:40:49.502890 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:40:49.502836    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-2kxls" podStartSLOduration=3.366668653 podStartE2EDuration="6.502819574s" totalImagesPullingTime="3.136150921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:49.501014869 +0000 UTC m=+355.986639462" watchObservedRunningTime="2026-07-28 12:40:49.502819574 +0000 UTC m=+355.988444237"
Jul 28 12:41:04.464908 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:41:04.464877    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-r2jsl"
Jul 28 12:41:10.491046 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:41:10.491014    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-2kxls"
Jul 28 12:42:02.305202 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.305167    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-cr5ch"]
Jul 28 12:42:02.307451 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.307433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.310582 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.310558    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:42:02.310729 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.310597    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 12:42:02.311101 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.311061    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:42:02.311950 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.311934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v9v8l\""
Jul 28 12:42:02.324770 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.324744    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cr5ch"]
Jul 28 12:42:02.338001 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.337978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0990dfe-051a-4be5-9fdc-c48654f75693-data\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.338129 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.338005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkvn\" (UniqueName: \"kubernetes.io/projected/c0990dfe-051a-4be5-9fdc-c48654f75693-kube-api-access-cwkvn\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.439295 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.439254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0990dfe-051a-4be5-9fdc-c48654f75693-data\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.439295 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.439296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkvn\" (UniqueName: \"kubernetes.io/projected/c0990dfe-051a-4be5-9fdc-c48654f75693-kube-api-access-cwkvn\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.439650 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.439627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0990dfe-051a-4be5-9fdc-c48654f75693-data\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.448318 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.448296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkvn\" (UniqueName: \"kubernetes.io/projected/c0990dfe-051a-4be5-9fdc-c48654f75693-kube-api-access-cwkvn\") pod \"seaweedfs-64568bcd49-cr5ch\" (UID: \"c0990dfe-051a-4be5-9fdc-c48654f75693\") " pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.616310 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.616275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:02.743915 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:02.743895    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cr5ch"]
Jul 28 12:42:02.745964 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:42:02.745935    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0990dfe_051a_4be5_9fdc_c48654f75693.slice/crio-2177a0eee8d3db837b57d3b74763c19776c15d06723d2fae5b41e40fac70d9c0 WatchSource:0}: Error finding container 2177a0eee8d3db837b57d3b74763c19776c15d06723d2fae5b41e40fac70d9c0: Status 404 returned error can't find the container with id 2177a0eee8d3db837b57d3b74763c19776c15d06723d2fae5b41e40fac70d9c0
Jul 28 12:42:03.749015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:03.748972    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cr5ch" event={"ID":"c0990dfe-051a-4be5-9fdc-c48654f75693","Type":"ContainerStarted","Data":"2177a0eee8d3db837b57d3b74763c19776c15d06723d2fae5b41e40fac70d9c0"}
Jul 28 12:42:05.758426 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:05.758391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cr5ch" event={"ID":"c0990dfe-051a-4be5-9fdc-c48654f75693","Type":"ContainerStarted","Data":"a38e54d4fca1c8cc9a94d0a834825e3e56ab5ca451385b273a210a9c51ec709b"}
Jul 28 12:42:05.758899 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:05.758529    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:42:05.775009 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:05.774959    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-cr5ch" podStartSLOduration=0.831596418 podStartE2EDuration="3.774942873s" totalImagesPullingTime="2.943346455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:05.774005464 +0000 UTC m=+432.259630056" watchObservedRunningTime="2026-07-28 12:42:05.774942873 +0000 UTC m=+432.260567465"
Jul 28 12:42:11.763821 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:42:11.763788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-cr5ch"
Jul 28 12:43:20.628603 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.628564    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-5l87j"]
Jul 28 12:43:20.631895 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.631871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.635054 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.635030    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 12:43:20.635176 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.635030    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xs8z9\""
Jul 28 12:43:20.643515 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.643496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-5l87j"]
Jul 28 12:43:20.646794 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.646776    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-98zz9"]
Jul 28 12:43:20.649669 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.649653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.652633 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.652610    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bk4gr\""
Jul 28 12:43:20.653040 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.653023    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 12:43:20.659689 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.659670    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-98zz9"]
Jul 28 12:43:20.729557 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.729525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.729764 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.729592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcw9\" (UniqueName: \"kubernetes.io/projected/953869a9-7494-4843-b179-fb4f94bf1dae-kube-api-access-9tcw9\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.729764 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.729619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953869a9-7494-4843-b179-fb4f94bf1dae-tls-certs\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.729764 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.729672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqml2\" (UniqueName: \"kubernetes.io/projected/11cba7c0-d129-4c6c-8737-12be323fa510-kube-api-access-vqml2\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.830354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.830312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqml2\" (UniqueName: \"kubernetes.io/projected/11cba7c0-d129-4c6c-8737-12be323fa510-kube-api-access-vqml2\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.830354 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.830360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.830604 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.830396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcw9\" (UniqueName: \"kubernetes.io/projected/953869a9-7494-4843-b179-fb4f94bf1dae-kube-api-access-9tcw9\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.830604 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.830414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953869a9-7494-4843-b179-fb4f94bf1dae-tls-certs\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.830604 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:43:20.830507    2558 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 12:43:20.830604 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:43:20.830577    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert podName:11cba7c0-d129-4c6c-8737-12be323fa510 nodeName:}" failed. No retries permitted until 2026-07-28 12:43:21.330559184 +0000 UTC m=+507.816183756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert") pod "odh-model-controller-75d9cdbddf-98zz9" (UID: "11cba7c0-d129-4c6c-8737-12be323fa510") : secret "odh-model-controller-webhook-cert" not found
Jul 28 12:43:20.833001 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.832979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953869a9-7494-4843-b179-fb4f94bf1dae-tls-certs\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.840985 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.840953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqml2\" (UniqueName: \"kubernetes.io/projected/11cba7c0-d129-4c6c-8737-12be323fa510-kube-api-access-vqml2\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:20.841343 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.841328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcw9\" (UniqueName: \"kubernetes.io/projected/953869a9-7494-4843-b179-fb4f94bf1dae-kube-api-access-9tcw9\") pod \"model-serving-api-7b954ffc95-5l87j\" (UID: \"953869a9-7494-4843-b179-fb4f94bf1dae\") " pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:20.944334 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:20.944238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:21.102464 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:21.102433    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-5l87j"]
Jul 28 12:43:21.105250 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:43:21.105222    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953869a9_7494_4843_b179_fb4f94bf1dae.slice/crio-55a4ca6630c94987444286ee477051a996ac283bbe244f25b228dbdbf3e132a9 WatchSource:0}: Error finding container 55a4ca6630c94987444286ee477051a996ac283bbe244f25b228dbdbf3e132a9: Status 404 returned error can't find the container with id 55a4ca6630c94987444286ee477051a996ac283bbe244f25b228dbdbf3e132a9
Jul 28 12:43:21.335075 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:21.334996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:21.337383 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:21.337364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11cba7c0-d129-4c6c-8737-12be323fa510-cert\") pod \"odh-model-controller-75d9cdbddf-98zz9\" (UID: \"11cba7c0-d129-4c6c-8737-12be323fa510\") " pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:21.560348 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:21.560318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:21.690927 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:21.690857    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-98zz9"]
Jul 28 12:43:21.692956 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:43:21.692932    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cba7c0_d129_4c6c_8737_12be323fa510.slice/crio-3f7abe739854e476f13092bc914ffa2b17af9849c36ab4cd1925bb66fed56d04 WatchSource:0}: Error finding container 3f7abe739854e476f13092bc914ffa2b17af9849c36ab4cd1925bb66fed56d04: Status 404 returned error can't find the container with id 3f7abe739854e476f13092bc914ffa2b17af9849c36ab4cd1925bb66fed56d04
Jul 28 12:43:22.052338 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:22.052256    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-5l87j" event={"ID":"953869a9-7494-4843-b179-fb4f94bf1dae","Type":"ContainerStarted","Data":"55a4ca6630c94987444286ee477051a996ac283bbe244f25b228dbdbf3e132a9"}
Jul 28 12:43:22.053322 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:22.053303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-98zz9" event={"ID":"11cba7c0-d129-4c6c-8737-12be323fa510","Type":"ContainerStarted","Data":"3f7abe739854e476f13092bc914ffa2b17af9849c36ab4cd1925bb66fed56d04"}
Jul 28 12:43:26.073601 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.073557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-98zz9" event={"ID":"11cba7c0-d129-4c6c-8737-12be323fa510","Type":"ContainerStarted","Data":"1fdd6fdbe1c7417788283bade846e8ae1739faa135b6aa53fe8e6dac16822287"}
Jul 28 12:43:26.074062 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.073625    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:26.074899 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.074876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-5l87j" event={"ID":"953869a9-7494-4843-b179-fb4f94bf1dae","Type":"ContainerStarted","Data":"42012dae748a4c7918db0f74fcad3723e24efa4223012119db344d78e45a4683"}
Jul 28 12:43:26.075015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.074998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:43:26.099315 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.099269    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-98zz9" podStartSLOduration=2.582601032 podStartE2EDuration="6.099253128s" totalImagesPullingTime="3.516652096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:26.098197556 +0000 UTC m=+512.583822149" watchObservedRunningTime="2026-07-28 12:43:26.099253128 +0000 UTC m=+512.584877719"
Jul 28 12:43:26.116301 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:26.116257    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-5l87j" podStartSLOduration=2.056909091 podStartE2EDuration="6.116244808s" totalImagesPullingTime="4.059335717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:26.115443958 +0000 UTC m=+512.601068551" watchObservedRunningTime="2026-07-28 12:43:26.116244808 +0000 UTC m=+512.601869400"
Jul 28 12:43:37.080776 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:37.080747    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-98zz9"
Jul 28 12:43:37.082793 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:43:37.082770    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-5l87j"
Jul 28 12:44:54.923455 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:54.923349    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:44:54.927458 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:54.924230    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:44:56.082650 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.082612    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:44:56.085798 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.085780    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.088478 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.088457    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 12:44:56.096474 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.096454    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:44:56.174072 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.174037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.174240 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.174079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtmk\" (UniqueName: \"kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.275252 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.275214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.275252 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.275254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtmk\" (UniqueName: \"kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.275592 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.275572    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.284015 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.283990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtmk\" (UniqueName: \"kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk\") pod \"seaweedfs-tls-custom-6644bf5d6c-krm6n\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.394446 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.394399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:44:56.521913 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:56.521888    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:44:56.524481 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:44:56.524450    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d5115c_5ad0_479e_bcc5_8c55fe4ac275.slice/crio-860e119e4c57cc883ffa9f9669e2e4ae14575024cc556db5818b35f58b5e4f7c WatchSource:0}: Error finding container 860e119e4c57cc883ffa9f9669e2e4ae14575024cc556db5818b35f58b5e4f7c: Status 404 returned error can't find the container with id 860e119e4c57cc883ffa9f9669e2e4ae14575024cc556db5818b35f58b5e4f7c
Jul 28 12:44:57.436295 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:57.436259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" event={"ID":"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275","Type":"ContainerStarted","Data":"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"}
Jul 28 12:44:57.436695 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:57.436312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" event={"ID":"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275","Type":"ContainerStarted","Data":"860e119e4c57cc883ffa9f9669e2e4ae14575024cc556db5818b35f58b5e4f7c"}
Jul 28 12:44:57.452331 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:57.452283    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" podStartSLOduration=1.16176787 podStartE2EDuration="1.452266275s" totalImagesPullingTime="290.498405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:57.450657365 +0000 UTC m=+603.936281956" watchObservedRunningTime="2026-07-28 12:44:57.452266275 +0000 UTC m=+603.937890866"
Jul 28 12:44:59.447156 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:59.447125    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:44:59.447559 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:44:59.447277    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" podUID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" containerName="seaweedfs-tls-custom" containerID="cri-o://d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d" gracePeriod=30
Jul 28 12:45:00.688398 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.688370    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:45:00.810199 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.810110    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data\") pod \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") "
Jul 28 12:45:00.810199 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.810175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmtmk\" (UniqueName: \"kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk\") pod \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\" (UID: \"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275\") "
Jul 28 12:45:00.811437 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.811405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data" (OuterVolumeSpecName: "data") pod "a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" (UID: "a1d5115c-5ad0-479e-bcc5-8c55fe4ac275"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:45:00.812483 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.812458    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk" (OuterVolumeSpecName: "kube-api-access-xmtmk") pod "a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" (UID: "a1d5115c-5ad0-479e-bcc5-8c55fe4ac275"). InnerVolumeSpecName "kube-api-access-xmtmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:45:00.911539 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.911494    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtmk\" (UniqueName: \"kubernetes.io/projected/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-kube-api-access-xmtmk\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:45:00.911539 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:00.911526    2558 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275-data\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 12:45:01.454732 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.454688    2558 generic.go:356] "Generic (PLEG): container finished" podID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" containerID="d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d" exitCode=0
Jul 28 12:45:01.454939 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.454747    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"
Jul 28 12:45:01.454939 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.454780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" event={"ID":"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275","Type":"ContainerDied","Data":"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"}
Jul 28 12:45:01.454939 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.454839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n" event={"ID":"a1d5115c-5ad0-479e-bcc5-8c55fe4ac275","Type":"ContainerDied","Data":"860e119e4c57cc883ffa9f9669e2e4ae14575024cc556db5818b35f58b5e4f7c"}
Jul 28 12:45:01.454939 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.454868    2558 scope.go:117] "RemoveContainer" containerID="d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"
Jul 28 12:45:01.463905 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.463885    2558 scope.go:117] "RemoveContainer" containerID="d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"
Jul 28 12:45:01.464151 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:01.464132    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d\": container with ID starting with d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d not found: ID does not exist" containerID="d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"
Jul 28 12:45:01.464191 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.464160    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d"} err="failed to get container status \"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d\": rpc error: code = NotFound desc = could not find container \"d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d\": container with ID starting with d19d7d2db78c2b1e477284518452d0e56525938c3480253147ee18f975373b0d not found: ID does not exist"
Jul 28 12:45:01.475274 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.475247    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:45:01.480949 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.480930    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-krm6n"]
Jul 28 12:45:01.509627 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.509605    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"]
Jul 28 12:45:01.509944 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.509931    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" containerName="seaweedfs-tls-custom"
Jul 28 12:45:01.509988 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.509946    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" containerName="seaweedfs-tls-custom"
Jul 28 12:45:01.510021 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.510008    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" containerName="seaweedfs-tls-custom"
Jul 28 12:45:01.514321 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.514306    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.517079 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.517058    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 12:45:01.517150 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.517083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 28 12:45:01.520626 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.520605    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"]
Jul 28 12:45:01.617563 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.617519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd650-b867-40e6-8811-042ae7f62a4c-data\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.617563 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.617558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwbz\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-kube-api-access-2bwbz\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.617798 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.617587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.718596 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.718523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd650-b867-40e6-8811-042ae7f62a4c-data\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.718596 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.718574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwbz\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-kube-api-access-2bwbz\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.719108 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.718623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.719108 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.719010    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b9bd650-b867-40e6-8811-042ae7f62a4c-data\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.721188 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.721164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.728009 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.727991    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwbz\" (UniqueName: \"kubernetes.io/projected/4b9bd650-b867-40e6-8811-042ae7f62a4c-kube-api-access-2bwbz\") pod \"seaweedfs-tls-custom-8b44d767d-8vqsp\" (UID: \"4b9bd650-b867-40e6-8811-042ae7f62a4c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.823453 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.823419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"
Jul 28 12:45:01.950567 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:01.950542    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8vqsp"]
Jul 28 12:45:01.952951 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:45:01.952922    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9bd650_b867_40e6_8811_042ae7f62a4c.slice/crio-7cda39914a54ebdb4800ea344bfc161837d9e0aa3fa88dfb97ff9059e3002346 WatchSource:0}: Error finding container 7cda39914a54ebdb4800ea344bfc161837d9e0aa3fa88dfb97ff9059e3002346: Status 404 returned error can't find the container with id 7cda39914a54ebdb4800ea344bfc161837d9e0aa3fa88dfb97ff9059e3002346
Jul 28 12:45:02.017911 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:02.017837    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d5115c-5ad0-479e-bcc5-8c55fe4ac275" path="/var/lib/kubelet/pods/a1d5115c-5ad0-479e-bcc5-8c55fe4ac275/volumes"
Jul 28 12:45:02.460786 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:02.460752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp" event={"ID":"4b9bd650-b867-40e6-8811-042ae7f62a4c","Type":"ContainerStarted","Data":"353e38abccd11cbce19a9a3136c40c8df9d96dc9f5d7e450ef9a429a5229fbb7"}
Jul 28 12:45:02.460976 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:02.460800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp" event={"ID":"4b9bd650-b867-40e6-8811-042ae7f62a4c","Type":"ContainerStarted","Data":"7cda39914a54ebdb4800ea344bfc161837d9e0aa3fa88dfb97ff9059e3002346"}
Jul 28 12:45:02.480659 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:02.480600    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-8vqsp" podStartSLOduration=1.199781152 podStartE2EDuration="1.480584411s" totalImagesPullingTime="280.803259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:45:02.478268056 +0000 UTC m=+608.963892647" watchObservedRunningTime="2026-07-28 12:45:02.480584411 +0000 UTC m=+608.966209003"
Jul 28 12:45:15.055023 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.054987    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"]
Jul 28 12:45:15.058509 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.058487    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.061301 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.061276    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 12:45:15.061301 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.061296    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 12:45:15.067127 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.067105    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"]
Jul 28 12:45:15.125321 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.125287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgg95\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-kube-api-access-wgg95\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.125509 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.125346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5aacb345-2e47-49c6-be35-16d7455d797d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.125509 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.125418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.226639 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.226604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.226842 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.226671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgg95\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-kube-api-access-wgg95\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.226842 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.226740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5aacb345-2e47-49c6-be35-16d7455d797d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.226842 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.226757    2558 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 28 12:45:15.226842 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.226775    2558 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh: secret "seaweedfs-tls-serving" not found
Jul 28 12:45:15.226842 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.226827    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving podName:5aacb345-2e47-49c6-be35-16d7455d797d nodeName:}" failed. No retries permitted until 2026-07-28 12:45:15.72681248 +0000 UTC m=+622.212437050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-h8mrh" (UID: "5aacb345-2e47-49c6-be35-16d7455d797d") : secret "seaweedfs-tls-serving" not found
Jul 28 12:45:15.227114 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.227087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5aacb345-2e47-49c6-be35-16d7455d797d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.238095 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.238073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgg95\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-kube-api-access-wgg95\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.730559 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:15.730470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:15.730763 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.730630    2558 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 28 12:45:15.730763 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.730648    2558 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh: secret "seaweedfs-tls-serving" not found
Jul 28 12:45:15.730763 ip-10-0-138-35 kubenswrapper[2558]: E0728 12:45:15.730715    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving podName:5aacb345-2e47-49c6-be35-16d7455d797d nodeName:}" failed. No retries permitted until 2026-07-28 12:45:16.730695891 +0000 UTC m=+623.216320463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-h8mrh" (UID: "5aacb345-2e47-49c6-be35-16d7455d797d") : secret "seaweedfs-tls-serving" not found
Jul 28 12:45:16.741387 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:16.741347    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:16.743887 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:16.743864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5aacb345-2e47-49c6-be35-16d7455d797d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-h8mrh\" (UID: \"5aacb345-2e47-49c6-be35-16d7455d797d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:16.870132 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:16.870098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"
Jul 28 12:45:16.997143 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:16.997071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh"]
Jul 28 12:45:17.000071 ip-10-0-138-35 kubenswrapper[2558]: W0728 12:45:17.000044    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aacb345_2e47_49c6_be35_16d7455d797d.slice/crio-3642aa6e50c99d18da7cb11704cf95402748efef17eb0f6645c247ea49b817d2 WatchSource:0}: Error finding container 3642aa6e50c99d18da7cb11704cf95402748efef17eb0f6645c247ea49b817d2: Status 404 returned error can't find the container with id 3642aa6e50c99d18da7cb11704cf95402748efef17eb0f6645c247ea49b817d2
Jul 28 12:45:17.530293 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:17.530199    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh" event={"ID":"5aacb345-2e47-49c6-be35-16d7455d797d","Type":"ContainerStarted","Data":"bfed7c8089cd71813f1898dd170b2bfc433789d37539d78a9eccdf21794ee5e2"}
Jul 28 12:45:17.530293 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:17.530254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh" event={"ID":"5aacb345-2e47-49c6-be35-16d7455d797d","Type":"ContainerStarted","Data":"3642aa6e50c99d18da7cb11704cf95402748efef17eb0f6645c247ea49b817d2"}
Jul 28 12:45:17.548593 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:45:17.548537    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-h8mrh" podStartSLOduration=2.287783905 podStartE2EDuration="2.548516839s" totalImagesPullingTime="260.732934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:45:17.5479607 +0000 UTC m=+624.033585291" watchObservedRunningTime="2026-07-28 12:45:17.548516839 +0000 UTC m=+624.034141430"
Jul 28 12:49:54.945006 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:49:54.944901    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:49:54.950845 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:49:54.947042    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:54:54.967377 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:54:54.967265    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:54:54.970345 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:54:54.969051    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:59:54.989209 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:59:54.989114    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 12:59:54.993060 ip-10-0-138-35 kubenswrapper[2558]: I0728 12:59:54.991193    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:04:55.011150 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:55.011037    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:04:55.016142 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:55.013101    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:04:59.647702 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.647664    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:04:59.650975 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.650959    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.653638 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.653620    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:04:59.653805 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.653779    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:04:59.653922 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.653844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:04:59.653922 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.653847    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 13:04:59.654908 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.654894    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:04:59.655211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.655194    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:04:59.655294 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.655197    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 13:04:59.660980 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.660959    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:04:59.717636 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.717601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.717636 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.717640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.717866 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.717662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.717866 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.717791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s4z\" (UniqueName: \"kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.717866 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.717842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818188 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s4z\" (UniqueName: \"kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818287    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818833 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.818984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.818961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.819066 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.819037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.820918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.820901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.827552 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.827531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s4z\" (UniqueName: \"kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-wp4ct\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:04:59.960946 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:04:59.960860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:05:00.093405 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:00.093380    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:05:00.095131 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:05:00.095103    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef282fd_318d_4aea_8884_8eaa0c2393bb.slice/crio-81657364391f9d2488647eb5c35d0ab51aa7c64081723a6fc905507edbb0a7e3 WatchSource:0}: Error finding container 81657364391f9d2488647eb5c35d0ab51aa7c64081723a6fc905507edbb0a7e3: Status 404 returned error can't find the container with id 81657364391f9d2488647eb5c35d0ab51aa7c64081723a6fc905507edbb0a7e3
Jul 28 13:05:00.097196 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:00.097177    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:05:00.227118 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:00.227030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerStarted","Data":"81657364391f9d2488647eb5c35d0ab51aa7c64081723a6fc905507edbb0a7e3"}
Jul 28 13:05:04.246937 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:04.246892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerStarted","Data":"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"}
Jul 28 13:05:08.263471 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:08.263382    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerID="9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3" exitCode=0
Jul 28 13:05:08.263471 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:08.263440    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerDied","Data":"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"}
Jul 28 13:05:23.344218 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:23.344173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerStarted","Data":"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"}
Jul 28 13:05:25.354345 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:25.354310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerStarted","Data":"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"}
Jul 28 13:05:25.354745 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:25.354450    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:05:25.383815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:25.383759    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podStartSLOduration=2.114653843 podStartE2EDuration="26.383738858s" totalImagesPullingTime="20.478870012s" totalInitContainerRuntime="3.790215003s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:25.382154449 +0000 UTC m=+1831.867779077" watchObservedRunningTime="2026-07-28 13:05:25.383738858 +0000 UTC m=+1831.869363459"
Jul 28 13:05:26.359140 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:26.359107    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:05:26.360322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:26.360297    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:05:27.362671 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:27.362623    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:05:32.366639 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:32.366607    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:05:32.367244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:32.367213    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:05:42.367619 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:42.367570    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:05:52.367436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:05:52.367393    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:06:02.367518 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:02.367472    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:06:12.367305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:12.367257    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:06:22.367950 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:22.367903    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:06:32.367157 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:32.367114    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 13:06:38.018354 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:38.018326    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:06:39.858029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:39.857995    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:06:39.858532 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:39.858262    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container" containerID="cri-o://54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2" gracePeriod=30
Jul 28 13:06:39.858532 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:39.858303    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kube-rbac-proxy" containerID="cri-o://f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc" gracePeriod=30
Jul 28 13:06:40.670276 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:40.670235    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerID="f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc" exitCode=2
Jul 28 13:06:40.670445 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:40.670286    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerDied","Data":"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"}
Jul 28 13:06:42.363325 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:42.363275    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 13:06:44.601121 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.601092    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:06:44.687497 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.687415    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerID="54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2" exitCode=0
Jul 28 13:06:44.687497 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.687466    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"
Jul 28 13:06:44.687683 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.687466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerDied","Data":"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"}
Jul 28 13:06:44.687683 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.687603    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct" event={"ID":"2ef282fd-318d-4aea-8884-8eaa0c2393bb","Type":"ContainerDied","Data":"81657364391f9d2488647eb5c35d0ab51aa7c64081723a6fc905507edbb0a7e3"}
Jul 28 13:06:44.687683 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.687634    2558 scope.go:117] "RemoveContainer" containerID="f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"
Jul 28 13:06:44.697384 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.697358    2558 scope.go:117] "RemoveContainer" containerID="54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"
Jul 28 13:06:44.705061 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.705044    2558 scope.go:117] "RemoveContainer" containerID="9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"
Jul 28 13:06:44.712205 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.712190    2558 scope.go:117] "RemoveContainer" containerID="f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"
Jul 28 13:06:44.712446 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:06:44.712425    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc\": container with ID starting with f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc not found: ID does not exist" containerID="f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"
Jul 28 13:06:44.712493 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.712461    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc"} err="failed to get container status \"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc\": rpc error: code = NotFound desc = could not find container \"f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc\": container with ID starting with f0f8ddbc8de0544fe1ede2d45f23358b00593987b75f4519b4c6c399baaf24dc not found: ID does not exist"
Jul 28 13:06:44.712493 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.712481    2558 scope.go:117] "RemoveContainer" containerID="54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"
Jul 28 13:06:44.712737 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:06:44.712705    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2\": container with ID starting with 54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2 not found: ID does not exist" containerID="54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"
Jul 28 13:06:44.712795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.712745    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2"} err="failed to get container status \"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2\": rpc error: code = NotFound desc = could not find container \"54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2\": container with ID starting with 54d52eee5a5690c889bde44bccafacd8367e714f6cdc8b9162462e883b43a2f2 not found: ID does not exist"
Jul 28 13:06:44.712795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.712764    2558 scope.go:117] "RemoveContainer" containerID="9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"
Jul 28 13:06:44.712993 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:06:44.712977    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3\": container with ID starting with 9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3 not found: ID does not exist" containerID="9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"
Jul 28 13:06:44.713036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.713000    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3"} err="failed to get container status \"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3\": rpc error: code = NotFound desc = could not find container \"9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3\": container with ID starting with 9138c0bb8ebbb41c17b7796a9ebccb1d66ddbe27943978e113d42b6660e51de3 not found: ID does not exist"
Jul 28 13:06:44.797329 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert\") pod \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") "
Jul 28 13:06:44.797490 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797343    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config\") pod \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") "
Jul 28 13:06:44.797490 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797377    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location\") pod \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") "
Jul 28 13:06:44.797490 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797401    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8s4z\" (UniqueName: \"kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z\") pod \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") "
Jul 28 13:06:44.797490 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797437    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls\") pod \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\" (UID: \"2ef282fd-318d-4aea-8884-8eaa0c2393bb\") "
Jul 28 13:06:44.797747 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797709    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ef282fd-318d-4aea-8884-8eaa0c2393bb" (UID: "2ef282fd-318d-4aea-8884-8eaa0c2393bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:44.797815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ef282fd-318d-4aea-8884-8eaa0c2393bb" (UID: "2ef282fd-318d-4aea-8884-8eaa0c2393bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:06:44.797815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.797777    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ef282fd-318d-4aea-8884-8eaa0c2393bb" (UID: "2ef282fd-318d-4aea-8884-8eaa0c2393bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:44.799688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.799663    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ef282fd-318d-4aea-8884-8eaa0c2393bb" (UID: "2ef282fd-318d-4aea-8884-8eaa0c2393bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:06:44.799798 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.799690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z" (OuterVolumeSpecName: "kube-api-access-b8s4z") pod "2ef282fd-318d-4aea-8884-8eaa0c2393bb" (UID: "2ef282fd-318d-4aea-8884-8eaa0c2393bb"). InnerVolumeSpecName "kube-api-access-b8s4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:06:44.898360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.898325    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:06:44.898360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.898359    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:06:44.898360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.898374    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:06:44.898360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.898382    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8s4z\" (UniqueName: \"kubernetes.io/projected/2ef282fd-318d-4aea-8884-8eaa0c2393bb-kube-api-access-b8s4z\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:06:44.898754 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:44.898392    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ef282fd-318d-4aea-8884-8eaa0c2393bb-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:06:45.007613 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007585    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:06:45.007940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007926    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kube-rbac-proxy"
Jul 28 13:06:45.007994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007942    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kube-rbac-proxy"
Jul 28 13:06:45.007994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007954    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="storage-initializer"
Jul 28 13:06:45.007994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007960    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="storage-initializer"
Jul 28 13:06:45.007994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007970    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container"
Jul 28 13:06:45.007994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.007976    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container"
Jul 28 13:06:45.008137 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.008021    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kserve-container"
Jul 28 13:06:45.008137 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.008028    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" containerName="kube-rbac-proxy"
Jul 28 13:06:45.012038 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.012018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.015252 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015230    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:06:45.015374 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:06:45.015506 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015488    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:06:45.015559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015516    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 13:06:45.015616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015586    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:06:45.015663 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015639    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:06:45.015746 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.015711    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:06:45.024366 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.024341    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:06:45.025904 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.025871    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:06:45.032603 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.032578    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-wp4ct"]
Jul 28 13:06:45.105628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.100617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.105628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.100670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.105628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.100712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.105628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.100800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdls\" (UniqueName: \"kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.105628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.100845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202047 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdls\" (UniqueName: \"kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202220 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202220 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202220 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202220 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202436 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:06:45.202261    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 13:06:45.202436 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:06:45.202319    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls podName:e87571da-1c8c-4a16-abab-76411f69fa11 nodeName:}" failed. No retries permitted until 2026-07-28 13:06:45.702301185 +0000 UTC m=+1912.187925755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls") pod "isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 13:06:45.202528 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202903 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.202995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.202974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.211773 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.211739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdls\" (UniqueName: \"kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.706249 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.706211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.708704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.708685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-sr884\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:45.922186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:45.922128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:46.017736 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:46.017683    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ef282fd-318d-4aea-8884-8eaa0c2393bb" path="/var/lib/kubelet/pods/2ef282fd-318d-4aea-8884-8eaa0c2393bb/volumes"
Jul 28 13:06:46.051093 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:46.051069    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:06:46.053126 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:06:46.053098    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87571da_1c8c_4a16_abab_76411f69fa11.slice/crio-23f8dff4550d6fc2d135b6829a0640286408885740e2fd32e34c5557e01cddea WatchSource:0}: Error finding container 23f8dff4550d6fc2d135b6829a0640286408885740e2fd32e34c5557e01cddea: Status 404 returned error can't find the container with id 23f8dff4550d6fc2d135b6829a0640286408885740e2fd32e34c5557e01cddea
Jul 28 13:06:46.698755 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:46.698701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerStarted","Data":"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"}
Jul 28 13:06:46.698936 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:46.698786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerStarted","Data":"23f8dff4550d6fc2d135b6829a0640286408885740e2fd32e34c5557e01cddea"}
Jul 28 13:06:49.712496 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:49.712461    2558 generic.go:356] "Generic (PLEG): container finished" podID="e87571da-1c8c-4a16-abab-76411f69fa11" containerID="ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda" exitCode=0
Jul 28 13:06:49.712894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:49.712540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerDied","Data":"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"}
Jul 28 13:06:50.718575 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.718538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerStarted","Data":"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"}
Jul 28 13:06:50.719003 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.718606    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerStarted","Data":"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"}
Jul 28 13:06:50.719003 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.718839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:50.719102 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.719012    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:50.720294 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.720266    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:06:50.737585 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:50.737541    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podStartSLOduration=3.320544082 podStartE2EDuration="6.737528306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.416984224s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:06:50.736319952 +0000 UTC m=+1917.221944544" watchObservedRunningTime="2026-07-28 13:06:50.737528306 +0000 UTC m=+1917.223152898"
Jul 28 13:06:51.723142 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:51.723100    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:06:56.727161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:56.727133    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:06:56.727641 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:06:56.727610    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:06.728059 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:06.728019    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:16.727548 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:16.727510    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:26.728202 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:26.728158    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:36.728495 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:36.728446    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:46.727988 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:46.727907    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:07:56.727853 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:07:56.727809    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:08:00.013109 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:00.013062    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 13:08:10.017389 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:10.017362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:08:15.057368 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.057331    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:08:15.057836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.057570    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container" containerID="cri-o://565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c" gracePeriod=30
Jul 28 13:08:15.057836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.057622    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kube-rbac-proxy" containerID="cri-o://967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670" gracePeriod=30
Jul 28 13:08:15.142854 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.142820    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:08:15.145163 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.145147    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.148193 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.148168    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:08:15.159278 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.159253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:08:15.215011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.214974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.215011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.215015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.215260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.215035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.215260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.215122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.215260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.215209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316246 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316246 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316246 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316465 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316465 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316755 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.316974 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.316941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.318896 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.318876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.325617 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.325585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8\") pod \"isvc-canary-rollback-predictor-ff46f797c-87z8f\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.459894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.459846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:15.587984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:15.587953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:08:15.589878 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:08:15.589850    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290013a0_b2c2_4a57_9083_7024d4861593.slice/crio-e3bb31c20355d7872d043bdba911540a377d154d202470d6e49d1889b6ba7697 WatchSource:0}: Error finding container e3bb31c20355d7872d043bdba911540a377d154d202470d6e49d1889b6ba7697: Status 404 returned error can't find the container with id e3bb31c20355d7872d043bdba911540a377d154d202470d6e49d1889b6ba7697
Jul 28 13:08:16.066517 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:16.066485    2558 generic.go:356] "Generic (PLEG): container finished" podID="e87571da-1c8c-4a16-abab-76411f69fa11" containerID="967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670" exitCode=2
Jul 28 13:08:16.066999 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:16.066553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerDied","Data":"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"}
Jul 28 13:08:16.068203 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:16.068180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerStarted","Data":"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"}
Jul 28 13:08:16.068340 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:16.068223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerStarted","Data":"e3bb31c20355d7872d043bdba911540a377d154d202470d6e49d1889b6ba7697"}
Jul 28 13:08:16.724251 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:16.724206    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 13:08:19.908457 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.908433    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:08:19.955444 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955367    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location\") pod \"e87571da-1c8c-4a16-abab-76411f69fa11\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") "
Jul 28 13:08:19.955444 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert\") pod \"e87571da-1c8c-4a16-abab-76411f69fa11\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") "
Jul 28 13:08:19.955623 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955467    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") pod \"e87571da-1c8c-4a16-abab-76411f69fa11\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") "
Jul 28 13:08:19.955623 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtdls\" (UniqueName: \"kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls\") pod \"e87571da-1c8c-4a16-abab-76411f69fa11\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") "
Jul 28 13:08:19.955623 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955511    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config\") pod \"e87571da-1c8c-4a16-abab-76411f69fa11\" (UID: \"e87571da-1c8c-4a16-abab-76411f69fa11\") "
Jul 28 13:08:19.955767 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955737    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e87571da-1c8c-4a16-abab-76411f69fa11" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:08:19.955906 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e87571da-1c8c-4a16-abab-76411f69fa11" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:19.955994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.955907    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e87571da-1c8c-4a16-abab-76411f69fa11" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:19.957653 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.957634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls" (OuterVolumeSpecName: "kube-api-access-qtdls") pod "e87571da-1c8c-4a16-abab-76411f69fa11" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11"). InnerVolumeSpecName "kube-api-access-qtdls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:19.957709 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:19.957676    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e87571da-1c8c-4a16-abab-76411f69fa11" (UID: "e87571da-1c8c-4a16-abab-76411f69fa11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:20.056835 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.056803    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e87571da-1c8c-4a16-abab-76411f69fa11-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:08:20.056835 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.056836    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:08:20.057021 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.056851    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e87571da-1c8c-4a16-abab-76411f69fa11-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:08:20.057021 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.056864    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtdls\" (UniqueName: \"kubernetes.io/projected/e87571da-1c8c-4a16-abab-76411f69fa11-kube-api-access-qtdls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:08:20.057021 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.056878    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e87571da-1c8c-4a16-abab-76411f69fa11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:08:20.086151 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.086116    2558 generic.go:356] "Generic (PLEG): container finished" podID="e87571da-1c8c-4a16-abab-76411f69fa11" containerID="565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c" exitCode=0
Jul 28 13:08:20.086326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.086196    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"
Jul 28 13:08:20.086326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.086201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerDied","Data":"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"}
Jul 28 13:08:20.086326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.086299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884" event={"ID":"e87571da-1c8c-4a16-abab-76411f69fa11","Type":"ContainerDied","Data":"23f8dff4550d6fc2d135b6829a0640286408885740e2fd32e34c5557e01cddea"}
Jul 28 13:08:20.086326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.086315    2558 scope.go:117] "RemoveContainer" containerID="967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"
Jul 28 13:08:20.087818 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.087794    2558 generic.go:356] "Generic (PLEG): container finished" podID="290013a0-b2c2-4a57-9083-7024d4861593" containerID="535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94" exitCode=0
Jul 28 13:08:20.087924 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.087899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerDied","Data":"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"}
Jul 28 13:08:20.094839 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.094824    2558 scope.go:117] "RemoveContainer" containerID="565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"
Jul 28 13:08:20.105760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.105736    2558 scope.go:117] "RemoveContainer" containerID="ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"
Jul 28 13:08:20.123469 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.123450    2558 scope.go:117] "RemoveContainer" containerID="967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"
Jul 28 13:08:20.123780 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.123760    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:08:20.123871 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:08:20.123771    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670\": container with ID starting with 967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670 not found: ID does not exist" containerID="967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"
Jul 28 13:08:20.123871 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.123803    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670"} err="failed to get container status \"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670\": rpc error: code = NotFound desc = could not find container \"967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670\": container with ID starting with 967201d6ba8cb5e1823c2ce0a835cd053bf370b08052fa4da475280ccf15b670 not found: ID does not exist"
Jul 28 13:08:20.123871 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.123827    2558 scope.go:117] "RemoveContainer" containerID="565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"
Jul 28 13:08:20.124147 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:08:20.124124    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c\": container with ID starting with 565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c not found: ID does not exist" containerID="565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"
Jul 28 13:08:20.124187 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.124152    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c"} err="failed to get container status \"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c\": rpc error: code = NotFound desc = could not find container \"565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c\": container with ID starting with 565f5dc64193070d8ae094a7311ff488326e44957874de8ab2e6cedcc778160c not found: ID does not exist"
Jul 28 13:08:20.124187 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.124166    2558 scope.go:117] "RemoveContainer" containerID="ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"
Jul 28 13:08:20.124396 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:08:20.124378    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda\": container with ID starting with ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda not found: ID does not exist" containerID="ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"
Jul 28 13:08:20.124458 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.124404    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda"} err="failed to get container status \"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda\": rpc error: code = NotFound desc = could not find container \"ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda\": container with ID starting with ff73841e35a586156a896897a2a859d99275378a2e93d8e690b4b38c2cbb3eda not found: ID does not exist"
Jul 28 13:08:20.128170 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:20.128151    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-sr884"]
Jul 28 13:08:21.095741 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.095680    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerStarted","Data":"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"}
Jul 28 13:08:21.096190 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.095775    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerStarted","Data":"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"}
Jul 28 13:08:21.096190 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.096148    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:21.096320 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.096194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:21.097454 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.097431    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:08:21.118864 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:21.118814    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podStartSLOduration=2.428739595 podStartE2EDuration="6.118799294s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690059699s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:21.116415438 +0000 UTC m=+2007.602040025" watchObservedRunningTime="2026-07-28 13:08:21.118799294 +0000 UTC m=+2007.604423952"
Jul 28 13:08:22.017852 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:22.017816    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" path="/var/lib/kubelet/pods/e87571da-1c8c-4a16-abab-76411f69fa11/volumes"
Jul 28 13:08:22.100004 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:22.099968    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:08:27.104322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:27.104290    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:08:27.104842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:27.104816    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:08:37.105429 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:37.105385    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:08:47.105608 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:47.105566    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:08:57.104950 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:08:57.104906    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:09:07.105234 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:07.105190    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:09:17.105167 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:17.105080    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:09:27.105669 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:27.105625    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 13:09:34.017309 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:34.017273    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:09:35.208810 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:35.208778    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:09:35.400389 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:35.400322    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container" containerID="cri-o://9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615" gracePeriod=30
Jul 28 13:09:35.400389 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:35.400364    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kube-rbac-proxy" containerID="cri-o://4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03" gracePeriod=30
Jul 28 13:09:36.406314 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:36.406285    2558 generic.go:356] "Generic (PLEG): container finished" podID="290013a0-b2c2-4a57-9083-7024d4861593" containerID="4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03" exitCode=2
Jul 28 13:09:36.406700 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:36.406355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerDied","Data":"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"}
Jul 28 13:09:37.100442 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:37.100402    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 13:09:40.147045 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.147017    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:09:40.239008 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.238933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8\") pod \"290013a0-b2c2-4a57-9083-7024d4861593\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") "
Jul 28 13:09:40.239008 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239007    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config\") pod \"290013a0-b2c2-4a57-9083-7024d4861593\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") "
Jul 28 13:09:40.239260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239050    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls\") pod \"290013a0-b2c2-4a57-9083-7024d4861593\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") "
Jul 28 13:09:40.239260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239067    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert\") pod \"290013a0-b2c2-4a57-9083-7024d4861593\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") "
Jul 28 13:09:40.239260 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239087    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location\") pod \"290013a0-b2c2-4a57-9083-7024d4861593\" (UID: \"290013a0-b2c2-4a57-9083-7024d4861593\") "
Jul 28 13:09:40.239458 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239399    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "290013a0-b2c2-4a57-9083-7024d4861593" (UID: "290013a0-b2c2-4a57-9083-7024d4861593"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:40.239573 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239542    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "290013a0-b2c2-4a57-9083-7024d4861593" (UID: "290013a0-b2c2-4a57-9083-7024d4861593"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:09:40.239637 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.239569    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "290013a0-b2c2-4a57-9083-7024d4861593" (UID: "290013a0-b2c2-4a57-9083-7024d4861593"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:40.241276 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.241246    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8" (OuterVolumeSpecName: "kube-api-access-tr5b8") pod "290013a0-b2c2-4a57-9083-7024d4861593" (UID: "290013a0-b2c2-4a57-9083-7024d4861593"). InnerVolumeSpecName "kube-api-access-tr5b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:09:40.241276 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.241269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "290013a0-b2c2-4a57-9083-7024d4861593" (UID: "290013a0-b2c2-4a57-9083-7024d4861593"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:09:40.339705 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.339677    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:09:40.339705 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.339703    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290013a0-b2c2-4a57-9083-7024d4861593-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:09:40.339951 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.339714    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/290013a0-b2c2-4a57-9083-7024d4861593-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:09:40.339951 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.339740    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/290013a0-b2c2-4a57-9083-7024d4861593-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:09:40.339951 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.339752    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5b8\" (UniqueName: \"kubernetes.io/projected/290013a0-b2c2-4a57-9083-7024d4861593-kube-api-access-tr5b8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:09:40.393086 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393048    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:09:40.393425 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393407    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393428    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393439    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="storage-initializer"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393448    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="storage-initializer"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393458    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kube-rbac-proxy"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393465    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kube-rbac-proxy"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393498    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="storage-initializer"
Jul 28 13:09:40.393509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393506    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="storage-initializer"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393516    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393524    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393533    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kube-rbac-proxy"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393541    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kube-rbac-proxy"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393626    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kserve-container"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393640    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87571da-1c8c-4a16-abab-76411f69fa11" containerName="kube-rbac-proxy"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393648    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kserve-container"
Jul 28 13:09:40.393863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.393656    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="290013a0-b2c2-4a57-9083-7024d4861593" containerName="kube-rbac-proxy"
Jul 28 13:09:40.397035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.397015    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.399612 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.399573    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 13:09:40.399768 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.399750    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 13:09:40.406304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.406271    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:09:40.425432 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.425400    2558 generic.go:356] "Generic (PLEG): container finished" podID="290013a0-b2c2-4a57-9083-7024d4861593" containerID="9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615" exitCode=0
Jul 28 13:09:40.425581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.425460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerDied","Data":"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"}
Jul 28 13:09:40.425581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.425470    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"
Jul 28 13:09:40.425581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.425527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f" event={"ID":"290013a0-b2c2-4a57-9083-7024d4861593","Type":"ContainerDied","Data":"e3bb31c20355d7872d043bdba911540a377d154d202470d6e49d1889b6ba7697"}
Jul 28 13:09:40.425581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.425563    2558 scope.go:117] "RemoveContainer" containerID="4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"
Jul 28 13:09:40.437865 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.437835    2558 scope.go:117] "RemoveContainer" containerID="9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"
Jul 28 13:09:40.442359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.442324    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.442359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.442375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7gw\" (UniqueName: \"kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.442525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.442421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.442525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.442462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.442525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.442499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.455396 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.455373    2558 scope.go:117] "RemoveContainer" containerID="535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"
Jul 28 13:09:40.461619 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.461588    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:09:40.463487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.463451    2558 scope.go:117] "RemoveContainer" containerID="4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"
Jul 28 13:09:40.463800 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:09:40.463765    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03\": container with ID starting with 4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03 not found: ID does not exist" containerID="4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"
Jul 28 13:09:40.463894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.463811    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03"} err="failed to get container status \"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03\": rpc error: code = NotFound desc = could not find container \"4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03\": container with ID starting with 4f98750c011264254dfd29ea9f3004d7c677960f8f967fb34f1544cf2a5dae03 not found: ID does not exist"
Jul 28 13:09:40.463894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.463850    2558 scope.go:117] "RemoveContainer" containerID="9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"
Jul 28 13:09:40.464233 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:09:40.464203    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615\": container with ID starting with 9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615 not found: ID does not exist" containerID="9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"
Jul 28 13:09:40.464320 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.464240    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615"} err="failed to get container status \"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615\": rpc error: code = NotFound desc = could not find container \"9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615\": container with ID starting with 9a99baeffe4f7a7b220a4e00e4e797bd39ee55d2104c02b62460ae088ece4615 not found: ID does not exist"
Jul 28 13:09:40.464320 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.464262    2558 scope.go:117] "RemoveContainer" containerID="535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"
Jul 28 13:09:40.464624 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:09:40.464602    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94\": container with ID starting with 535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94 not found: ID does not exist" containerID="535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"
Jul 28 13:09:40.464696 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.464666    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94"} err="failed to get container status \"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94\": rpc error: code = NotFound desc = could not find container \"535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94\": container with ID starting with 535aac4a4fd6fbce7bf5d927576262b062c8a324613a87b030b72c841eab5f94 not found: ID does not exist"
Jul 28 13:09:40.466023 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.465998    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-87z8f"]
Jul 28 13:09:40.543256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7gw\" (UniqueName: \"kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543617 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543777 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543752    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.543995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.543975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.544066 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.544042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.545733 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.545700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.551022 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.551006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7gw\" (UniqueName: \"kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw\") pod \"isvc-canary-stop-predictor-59777d5f7c-6cj5n\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.710247 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.710204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:40.842487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:40.842413    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:09:40.845577 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:09:40.845552    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef11ce9e_32e6_47dd_8788_d9135d4a6b4f.slice/crio-21808a65a36267bb7265268b679dbcefcf4f2ac654189c87a6b4d1099e823e09 WatchSource:0}: Error finding container 21808a65a36267bb7265268b679dbcefcf4f2ac654189c87a6b4d1099e823e09: Status 404 returned error can't find the container with id 21808a65a36267bb7265268b679dbcefcf4f2ac654189c87a6b4d1099e823e09
Jul 28 13:09:41.430882 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:41.430838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerStarted","Data":"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"}
Jul 28 13:09:41.431321 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:41.430921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerStarted","Data":"21808a65a36267bb7265268b679dbcefcf4f2ac654189c87a6b4d1099e823e09"}
Jul 28 13:09:42.017804 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:42.017770    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290013a0-b2c2-4a57-9083-7024d4861593" path="/var/lib/kubelet/pods/290013a0-b2c2-4a57-9083-7024d4861593/volumes"
Jul 28 13:09:45.450101 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:45.450071    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerID="f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384" exitCode=0
Jul 28 13:09:45.450483 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:45.450153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerDied","Data":"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"}
Jul 28 13:09:46.456023 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:46.455990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerStarted","Data":"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"}
Jul 28 13:09:46.456376 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:46.456051    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerStarted","Data":"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"}
Jul 28 13:09:46.456376 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:46.456234    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:46.476360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:46.476315    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podStartSLOduration=2.69928169 podStartE2EDuration="6.47630082s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77701913s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:09:46.47536235 +0000 UTC m=+2092.960986945" watchObservedRunningTime="2026-07-28 13:09:46.47630082 +0000 UTC m=+2092.961925411"
Jul 28 13:09:47.460068 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:47.460037    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:47.461430 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:47.461404    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:09:48.463578 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:48.463541    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:09:53.468112 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:53.468080    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:09:53.468606 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:53.468578    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:09:55.034194 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:55.034089    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:09:55.061432 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:09:55.038531    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:10:03.468525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:03.468486    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:10:13.468787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:13.468740    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:10:23.468787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:23.468746    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:10:33.469024 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:33.468977    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:10:43.468904 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:43.468861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:10:53.468763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:10:53.468663    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:11:03.469359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:03.469316    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:11:15.441708 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:15.441652    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:11:15.442271 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:15.441965    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container" containerID="cri-o://d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57" gracePeriod=30
Jul 28 13:11:15.442700 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:15.442023    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kube-rbac-proxy" containerID="cri-o://7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961" gracePeriod=30
Jul 28 13:11:15.828090 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:15.828004    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerID="7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961" exitCode=2
Jul 28 13:11:15.828090 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:15.828077    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerDied","Data":"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"}
Jul 28 13:11:18.464274 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:18.464227    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 13:11:20.185083 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.185058    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:11:20.266304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266213    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config\") pod \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") "
Jul 28 13:11:20.266304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls\") pod \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") "
Jul 28 13:11:20.266304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266296    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert\") pod \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") "
Jul 28 13:11:20.266593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv7gw\" (UniqueName: \"kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw\") pod \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") "
Jul 28 13:11:20.266593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266402    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location\") pod \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\" (UID: \"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f\") "
Jul 28 13:11:20.266783 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" (UID: "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:20.266908 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" (UID: "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:20.266908 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.266812    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" (UID: "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:11:20.268513 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.268492    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" (UID: "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:11:20.268620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.268601    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw" (OuterVolumeSpecName: "kube-api-access-cv7gw") pod "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" (UID: "ef11ce9e-32e6-47dd-8788-d9135d4a6b4f"). InnerVolumeSpecName "kube-api-access-cv7gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:11:20.367385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.367345    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv7gw\" (UniqueName: \"kubernetes.io/projected/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-api-access-cv7gw\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:11:20.367385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.367377    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:11:20.367385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.367386    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:11:20.367599 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.367396    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:11:20.367599 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.367407    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:11:20.851453 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.851421    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerID="d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57" exitCode=0
Jul 28 13:11:20.851652 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.851474    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"
Jul 28 13:11:20.851652 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.851514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerDied","Data":"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"}
Jul 28 13:11:20.851652 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.851585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n" event={"ID":"ef11ce9e-32e6-47dd-8788-d9135d4a6b4f","Type":"ContainerDied","Data":"21808a65a36267bb7265268b679dbcefcf4f2ac654189c87a6b4d1099e823e09"}
Jul 28 13:11:20.851652 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.851622    2558 scope.go:117] "RemoveContainer" containerID="7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"
Jul 28 13:11:20.860290 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.860271    2558 scope.go:117] "RemoveContainer" containerID="d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"
Jul 28 13:11:20.867912 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.867892    2558 scope.go:117] "RemoveContainer" containerID="f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"
Jul 28 13:11:20.873789 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.873699    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:11:20.875761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.875745    2558 scope.go:117] "RemoveContainer" containerID="7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"
Jul 28 13:11:20.876012 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:11:20.875994    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961\": container with ID starting with 7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961 not found: ID does not exist" containerID="7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"
Jul 28 13:11:20.876074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.876022    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961"} err="failed to get container status \"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961\": rpc error: code = NotFound desc = could not find container \"7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961\": container with ID starting with 7a41c95cd949165dcc2eb9e5a72fc22dacd489ebc541fd3f07705297f801f961 not found: ID does not exist"
Jul 28 13:11:20.876074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.876039    2558 scope.go:117] "RemoveContainer" containerID="d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"
Jul 28 13:11:20.876266 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:11:20.876250    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57\": container with ID starting with d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57 not found: ID does not exist" containerID="d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"
Jul 28 13:11:20.876305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.876273    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57"} err="failed to get container status \"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57\": rpc error: code = NotFound desc = could not find container \"d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57\": container with ID starting with d9532d64540a26bf49f3e402c4168e6238bb84b79f6cba8b10eaf4ba70a84f57 not found: ID does not exist"
Jul 28 13:11:20.876305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.876288    2558 scope.go:117] "RemoveContainer" containerID="f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"
Jul 28 13:11:20.876521 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:11:20.876483    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384\": container with ID starting with f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384 not found: ID does not exist" containerID="f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"
Jul 28 13:11:20.876587 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.876525    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384"} err="failed to get container status \"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384\": rpc error: code = NotFound desc = could not find container \"f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384\": container with ID starting with f7a7f6fe2ddc7d04fff51d0e47c6f6a2208e7b181f0a96cd787c3dd673d96384 not found: ID does not exist"
Jul 28 13:11:20.878834 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:20.878814    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-6cj5n"]
Jul 28 13:11:22.017504 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:11:22.017460    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" path="/var/lib/kubelet/pods/ef11ce9e-32e6-47dd-8788-d9135d4a6b4f/volumes"
Jul 28 13:14:55.062096 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:14:55.061986    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:14:55.067153 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:14:55.067134    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:18:01.958573 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958534    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958864    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958877    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958891    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kube-rbac-proxy"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958900    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kube-rbac-proxy"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958915    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="storage-initializer"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958921    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="storage-initializer"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958980    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kube-rbac-proxy"
Jul 28 13:18:01.959062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.958989    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef11ce9e-32e6-47dd-8788-d9135d4a6b4f" containerName="kserve-container"
Jul 28 13:18:01.961276 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.961260    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:01.964107 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.964084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:18:01.964248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.964084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:18:01.964248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.964084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:18:01.964248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.964094    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:18:01.965299 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.965273    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:18:01.965299 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.965270    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 13:18:01.965440 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.965330    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:18:01.972675 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:01.972650    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:02.043142 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.043110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.043305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.043143    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.043305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.043182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.043305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.043271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.043401 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.043306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czns8\" (UniqueName: \"kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.143762 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.143687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.143762 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.143764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144044 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.143792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144044 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.143848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144044 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.143871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czns8\" (UniqueName: \"kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144259 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.144231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144469 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.144446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.144538 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.144458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.146349 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.146332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.152999 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.152970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czns8\" (UniqueName: \"kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.272531 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.272438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:02.401935 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.401909    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:02.404243 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:18:02.404217    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37bbb3a_f98b_4618_892f_c4b7b99c8c72.slice/crio-534b69e2d69a7a7718b271c7a48ee517e1ca0bafbf9561e441304bef72689f10 WatchSource:0}: Error finding container 534b69e2d69a7a7718b271c7a48ee517e1ca0bafbf9561e441304bef72689f10: Status 404 returned error can't find the container with id 534b69e2d69a7a7718b271c7a48ee517e1ca0bafbf9561e441304bef72689f10
Jul 28 13:18:02.463921 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.463891    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerStarted","Data":"77c49f98ed8544f7e158374dbb9568e38545536d6c44e0ba13d047b4dc0b85c9"}
Jul 28 13:18:02.464036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:02.463949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerStarted","Data":"534b69e2d69a7a7718b271c7a48ee517e1ca0bafbf9561e441304bef72689f10"}
Jul 28 13:18:06.482610 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:06.482572    2558 generic.go:356] "Generic (PLEG): container finished" podID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerID="77c49f98ed8544f7e158374dbb9568e38545536d6c44e0ba13d047b4dc0b85c9" exitCode=0
Jul 28 13:18:06.482979 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:06.482647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerDied","Data":"77c49f98ed8544f7e158374dbb9568e38545536d6c44e0ba13d047b4dc0b85c9"}
Jul 28 13:18:06.483976 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:06.483958    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:18:13.522656 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:13.522622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerStarted","Data":"51907bda7e5ee022b38d4e710c9c557ebe2c89e4b5f2e15e4f4611ebea687f76"}
Jul 28 13:18:13.523075 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:13.522694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerStarted","Data":"77c8a847ea6cc51dbfa6d7696c923572e24a74c0ca028764c97040d448261c81"}
Jul 28 13:18:13.523075 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:13.522893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:13.544613 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:13.544563    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" podStartSLOduration=2.414150756 podStartE2EDuration="12.54454837s" totalImagesPullingTime="6.354305455s" totalInitContainerRuntime="3.776092159s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:18:13.542179994 +0000 UTC m=+2600.027804585" watchObservedRunningTime="2026-07-28 13:18:13.54454837 +0000 UTC m=+2600.030172962"
Jul 28 13:18:14.527358 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:14.527314    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:14.528769 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:14.528744    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 13:18:15.531752 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:15.531645    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 13:18:20.536044 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:20.536010    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:20.536875 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:20.536854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:22.017631 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.017603    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:22.018011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.017824    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container" containerID="cri-o://77c8a847ea6cc51dbfa6d7696c923572e24a74c0ca028764c97040d448261c81" gracePeriod=30
Jul 28 13:18:22.018011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.017838    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kube-rbac-proxy" containerID="cri-o://51907bda7e5ee022b38d4e710c9c557ebe2c89e4b5f2e15e4f4611ebea687f76" gracePeriod=30
Jul 28 13:18:22.563574 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.563532    2558 generic.go:356] "Generic (PLEG): container finished" podID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerID="51907bda7e5ee022b38d4e710c9c557ebe2c89e4b5f2e15e4f4611ebea687f76" exitCode=2
Jul 28 13:18:22.563574 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.563573    2558 generic.go:356] "Generic (PLEG): container finished" podID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerID="77c8a847ea6cc51dbfa6d7696c923572e24a74c0ca028764c97040d448261c81" exitCode=0
Jul 28 13:18:22.563845 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.563596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerDied","Data":"51907bda7e5ee022b38d4e710c9c557ebe2c89e4b5f2e15e4f4611ebea687f76"}
Jul 28 13:18:22.563845 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.563687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerDied","Data":"77c8a847ea6cc51dbfa6d7696c923572e24a74c0ca028764c97040d448261c81"}
Jul 28 13:18:22.673645 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.673622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:22.701942 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.701911    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls\") pod \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") "
Jul 28 13:18:22.701942 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.701949    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location\") pod \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") "
Jul 28 13:18:22.702166 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.701984    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert\") pod \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") "
Jul 28 13:18:22.702166 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.702007    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czns8\" (UniqueName: \"kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8\") pod \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") "
Jul 28 13:18:22.702166 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.702028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config\") pod \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\" (UID: \"a37bbb3a-f98b-4618-892f-c4b7b99c8c72\") "
Jul 28 13:18:22.702375 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.702345    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a37bbb3a-f98b-4618-892f-c4b7b99c8c72" (UID: "a37bbb3a-f98b-4618-892f-c4b7b99c8c72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:18:22.702443 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.702391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a37bbb3a-f98b-4618-892f-c4b7b99c8c72" (UID: "a37bbb3a-f98b-4618-892f-c4b7b99c8c72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:18:22.702443 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.702411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a37bbb3a-f98b-4618-892f-c4b7b99c8c72" (UID: "a37bbb3a-f98b-4618-892f-c4b7b99c8c72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:18:22.704201 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.704179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a37bbb3a-f98b-4618-892f-c4b7b99c8c72" (UID: "a37bbb3a-f98b-4618-892f-c4b7b99c8c72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:18:22.704201 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.704181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8" (OuterVolumeSpecName: "kube-api-access-czns8") pod "a37bbb3a-f98b-4618-892f-c4b7b99c8c72" (UID: "a37bbb3a-f98b-4618-892f-c4b7b99c8c72"). InnerVolumeSpecName "kube-api-access-czns8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:18:22.803035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.802951    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:18:22.803035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.802980    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:18:22.803035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.802989    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:18:22.803035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.803008    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czns8\" (UniqueName: \"kubernetes.io/projected/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-api-access-czns8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:18:22.803035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:22.803027    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37bbb3a-f98b-4618-892f-c4b7b99c8c72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:18:23.568953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.568915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275" event={"ID":"a37bbb3a-f98b-4618-892f-c4b7b99c8c72","Type":"ContainerDied","Data":"534b69e2d69a7a7718b271c7a48ee517e1ca0bafbf9561e441304bef72689f10"}
Jul 28 13:18:23.569355 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.568961    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"
Jul 28 13:18:23.569355 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.569000    2558 scope.go:117] "RemoveContainer" containerID="51907bda7e5ee022b38d4e710c9c557ebe2c89e4b5f2e15e4f4611ebea687f76"
Jul 28 13:18:23.577322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.577306    2558 scope.go:117] "RemoveContainer" containerID="77c8a847ea6cc51dbfa6d7696c923572e24a74c0ca028764c97040d448261c81"
Jul 28 13:18:23.584545 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.584522    2558 scope.go:117] "RemoveContainer" containerID="77c49f98ed8544f7e158374dbb9568e38545536d6c44e0ba13d047b4dc0b85c9"
Jul 28 13:18:23.591048 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.591019    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:23.596914 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:23.596891    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-vp275"]
Jul 28 13:18:24.018415 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:18:24.018381    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" path="/var/lib/kubelet/pods/a37bbb3a-f98b-4618-892f-c4b7b99c8c72/volumes"
Jul 28 13:19:55.085437 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:19:55.085328    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:19:55.091664 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:19:55.091643    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:20:47.738093 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738053    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738403    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="storage-initializer"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738417    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="storage-initializer"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738431    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738436    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738444    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kube-rbac-proxy"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738454    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kube-rbac-proxy"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738504    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kserve-container"
Jul 28 13:20:47.738530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.738513    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37bbb3a-f98b-4618-892f-c4b7b99c8c72" containerName="kube-rbac-proxy"
Jul 28 13:20:47.740611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.740591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.743245 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.743221    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:20:47.743364 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.743343    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 13:20:47.743426 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.743360    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:20:47.743426 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.743360    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:20:47.744346 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.744319    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:20:47.744467 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.744345    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 13:20:47.744467 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.744323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:20:47.751604 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.751581    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:20:47.840763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.840706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.840940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.840770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppmd\" (UniqueName: \"kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.840940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.840793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.840940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.840814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.840940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.840860    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.941677 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.941639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.941892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.941696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.941892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.941822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.941892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.941862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppmd\" (UniqueName: \"kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.941892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.941885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.942110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.942092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.942562 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.942541    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.942605 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.942541    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.944300 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.944282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:47.950808 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:47.950784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppmd\" (UniqueName: \"kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bk8k9\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:48.050781 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:48.050676    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:20:48.180686 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:48.180596    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:20:48.182996 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:20:48.182965    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c112ec_0791_4dfb_9728_fc14749ea1a6.slice/crio-d9759600075897fc97054ff4a84160648dd697f9da8979b66cf839707c73a69f WatchSource:0}: Error finding container d9759600075897fc97054ff4a84160648dd697f9da8979b66cf839707c73a69f: Status 404 returned error can't find the container with id d9759600075897fc97054ff4a84160648dd697f9da8979b66cf839707c73a69f
Jul 28 13:20:49.152022 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:49.151982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerStarted","Data":"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"}
Jul 28 13:20:49.152387 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:49.152058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerStarted","Data":"d9759600075897fc97054ff4a84160648dd697f9da8979b66cf839707c73a69f"}
Jul 28 13:20:53.170487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:53.170454    2558 generic.go:356] "Generic (PLEG): container finished" podID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerID="20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8" exitCode=0
Jul 28 13:20:53.170924 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:20:53.170514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerDied","Data":"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"}
Jul 28 13:21:05.225555 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.225517    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerStarted","Data":"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"}
Jul 28 13:21:05.225981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.225585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerStarted","Data":"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"}
Jul 28 13:21:05.225981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.225916    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:21:05.226075 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.226055    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:21:05.227304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.227279    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:05.246892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:05.246839    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podStartSLOduration=2.354807706 podStartE2EDuration="18.246820367s" totalImagesPullingTime="11.475116446s" totalInitContainerRuntime="4.416896215s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:21:05.24630246 +0000 UTC m=+2771.731927053" watchObservedRunningTime="2026-07-28 13:21:05.246820367 +0000 UTC m=+2771.732444958"
Jul 28 13:21:06.230146 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:06.230108    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:11.235252 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:11.235213    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:21:11.235758 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:11.235709    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:21.236692 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:21.236645    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:31.236116 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:31.236077    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:41.236096 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:41.236056    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:21:51.235896 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:51.235864    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:21:59.362586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:59.362541    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:21:59.363090 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:59.362868    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" containerID="cri-o://9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2" gracePeriod=30
Jul 28 13:21:59.363090 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:21:59.362911    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kube-rbac-proxy" containerID="cri-o://d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822" gracePeriod=30
Jul 28 13:22:00.453514 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:00.453477    2558 generic.go:356] "Generic (PLEG): container finished" podID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerID="d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822" exitCode=2
Jul 28 13:22:00.453902 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:00.453539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerDied","Data":"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"}
Jul 28 13:22:01.231030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:01.230990    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 13:22:01.236486 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:01.236462    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:22:02.109266 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.109241    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:22:02.137203 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137165    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location\") pod \"55c112ec-0791-4dfb-9728-fc14749ea1a6\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") "
Jul 28 13:22:02.137356 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config\") pod \"55c112ec-0791-4dfb-9728-fc14749ea1a6\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") "
Jul 28 13:22:02.137356 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137295    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert\") pod \"55c112ec-0791-4dfb-9728-fc14749ea1a6\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") "
Jul 28 13:22:02.137356 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137352    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ppmd\" (UniqueName: \"kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd\") pod \"55c112ec-0791-4dfb-9728-fc14749ea1a6\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") "
Jul 28 13:22:02.137515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls\") pod \"55c112ec-0791-4dfb-9728-fc14749ea1a6\" (UID: \"55c112ec-0791-4dfb-9728-fc14749ea1a6\") "
Jul 28 13:22:02.137874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55c112ec-0791-4dfb-9728-fc14749ea1a6" (UID: "55c112ec-0791-4dfb-9728-fc14749ea1a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:22:02.137874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137676    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:22:02.137874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.137748    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55c112ec-0791-4dfb-9728-fc14749ea1a6" (UID: "55c112ec-0791-4dfb-9728-fc14749ea1a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:22:02.140025 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.139998    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd" (OuterVolumeSpecName: "kube-api-access-2ppmd") pod "55c112ec-0791-4dfb-9728-fc14749ea1a6" (UID: "55c112ec-0791-4dfb-9728-fc14749ea1a6"). InnerVolumeSpecName "kube-api-access-2ppmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:22:02.140342 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.140316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55c112ec-0791-4dfb-9728-fc14749ea1a6" (UID: "55c112ec-0791-4dfb-9728-fc14749ea1a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:22:02.147474 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.147391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55c112ec-0791-4dfb-9728-fc14749ea1a6" (UID: "55c112ec-0791-4dfb-9728-fc14749ea1a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:22:02.238670 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.238633    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppmd\" (UniqueName: \"kubernetes.io/projected/55c112ec-0791-4dfb-9728-fc14749ea1a6-kube-api-access-2ppmd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:22:02.238670 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.238668    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c112ec-0791-4dfb-9728-fc14749ea1a6-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:22:02.238879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.238684    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c112ec-0791-4dfb-9728-fc14749ea1a6-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:22:02.238879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.238696    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c112ec-0791-4dfb-9728-fc14749ea1a6-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:22:02.463711 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.463620    2558 generic.go:356] "Generic (PLEG): container finished" podID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerID="9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2" exitCode=0
Jul 28 13:22:02.463711 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.463666    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerDied","Data":"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"}
Jul 28 13:22:02.463962 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.463739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9" event={"ID":"55c112ec-0791-4dfb-9728-fc14749ea1a6","Type":"ContainerDied","Data":"d9759600075897fc97054ff4a84160648dd697f9da8979b66cf839707c73a69f"}
Jul 28 13:22:02.463962 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.463744    2558 scope.go:117] "RemoveContainer" containerID="d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"
Jul 28 13:22:02.463962 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.463759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"
Jul 28 13:22:02.472255 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.472240    2558 scope.go:117] "RemoveContainer" containerID="9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"
Jul 28 13:22:02.479537 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.479519    2558 scope.go:117] "RemoveContainer" containerID="20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"
Jul 28 13:22:02.484388 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.484368    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:22:02.487395 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.487381    2558 scope.go:117] "RemoveContainer" containerID="d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"
Jul 28 13:22:02.487667 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:22:02.487647    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822\": container with ID starting with d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822 not found: ID does not exist" containerID="d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"
Jul 28 13:22:02.487760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.487674    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822"} err="failed to get container status \"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822\": rpc error: code = NotFound desc = could not find container \"d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822\": container with ID starting with d9a08e7e92e7bd4c9ef988f4d6a71814afeb8b8bc378453913e30d2bf0e28822 not found: ID does not exist"
Jul 28 13:22:02.487760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.487692    2558 scope.go:117] "RemoveContainer" containerID="9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"
Jul 28 13:22:02.487999 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:22:02.487980    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2\": container with ID starting with 9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2 not found: ID does not exist" containerID="9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"
Jul 28 13:22:02.488057 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.488005    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2"} err="failed to get container status \"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2\": rpc error: code = NotFound desc = could not find container \"9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2\": container with ID starting with 9a35f1dc5af511e3a65b87ef5c59f0b58fdeb4c5dd48914dfe8c17fbc40a4fe2 not found: ID does not exist"
Jul 28 13:22:02.488057 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.488020    2558 scope.go:117] "RemoveContainer" containerID="20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"
Jul 28 13:22:02.488243 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:22:02.488225    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8\": container with ID starting with 20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8 not found: ID does not exist" containerID="20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"
Jul 28 13:22:02.488303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.488248    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8"} err="failed to get container status \"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8\": rpc error: code = NotFound desc = could not find container \"20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8\": container with ID starting with 20a15b70f3916f0ee9ef58aff12bedea113efcbcb3ff42ed57c4a814259a73e8 not found: ID does not exist"
Jul 28 13:22:02.488303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:02.488256    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bk8k9"]
Jul 28 13:22:04.017522 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.017487    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" path="/var/lib/kubelet/pods/55c112ec-0791-4dfb-9728-fc14749ea1a6/volumes"
Jul 28 13:22:04.507366 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507333    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:22:04.507646 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507632    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container"
Jul 28 13:22:04.507698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507650    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container"
Jul 28 13:22:04.507698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507661    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kube-rbac-proxy"
Jul 28 13:22:04.507698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507666    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kube-rbac-proxy"
Jul 28 13:22:04.507698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507674    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="storage-initializer"
Jul 28 13:22:04.507698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507680    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="storage-initializer"
Jul 28 13:22:04.507863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507761    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kserve-container"
Jul 28 13:22:04.507863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.507768    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c112ec-0791-4dfb-9728-fc14749ea1a6" containerName="kube-rbac-proxy"
Jul 28 13:22:04.510379 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.510356    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.513009 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.512981    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:22:04.513160 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.513035    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:22:04.513160 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.513081    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:22:04.513386 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.513369    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:22:04.513455 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.513404    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:22:04.514396 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.514380    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:22:04.514491 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.514417    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 13:22:04.522836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.522814    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:22:04.559294 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.559257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.559294 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.559297    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.559503 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.559336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7n4x\" (UniqueName: \"kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.559503 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.559433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.559503 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.559468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.660764 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.660711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.660764 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.660762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.660802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7n4x\" (UniqueName: \"kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.660849    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.660871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661029 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:22:04.660976    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 13:22:04.661029 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:22:04.661033    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls podName:f48ea81b-194e-4540-8249-9ecc34357191 nodeName:}" failed. No retries permitted until 2026-07-28 13:22:05.161018044 +0000 UTC m=+2831.646642615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-dq6zj" (UID: "f48ea81b-194e-4540-8249-9ecc34357191") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 13:22:04.661292 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.661236    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661503 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.661480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.661503 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.661492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:04.673073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:04.673040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7n4x\" (UniqueName: \"kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:05.166009 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:05.165973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:05.168538 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:05.168514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-dq6zj\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:05.424304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:05.424208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:05.557194 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:05.557171    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:22:05.559358 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:22:05.559317    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48ea81b_194e_4540_8249_9ecc34357191.slice/crio-4996458a40ea02416d48fad6d7ca1462e15310965c45c30718ae169eb9ffa79b WatchSource:0}: Error finding container 4996458a40ea02416d48fad6d7ca1462e15310965c45c30718ae169eb9ffa79b: Status 404 returned error can't find the container with id 4996458a40ea02416d48fad6d7ca1462e15310965c45c30718ae169eb9ffa79b
Jul 28 13:22:06.485236 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:06.485200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerStarted","Data":"bc463f109be9f917a8ad58608ab4ba09c6c47b6fb4ddf2d307013b4d026b2ca2"}
Jul 28 13:22:06.485618 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:06.485261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerStarted","Data":"4996458a40ea02416d48fad6d7ca1462e15310965c45c30718ae169eb9ffa79b"}
Jul 28 13:22:11.509691 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:11.509655    2558 generic.go:356] "Generic (PLEG): container finished" podID="f48ea81b-194e-4540-8249-9ecc34357191" containerID="bc463f109be9f917a8ad58608ab4ba09c6c47b6fb4ddf2d307013b4d026b2ca2" exitCode=0
Jul 28 13:22:11.510097 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:11.509748    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerDied","Data":"bc463f109be9f917a8ad58608ab4ba09c6c47b6fb4ddf2d307013b4d026b2ca2"}
Jul 28 13:22:12.515485 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.515447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerStarted","Data":"eb978e310d9c2f8ff84dadf368f98fa5c9013a168bd4a1f72f17309d3ac9e2a9"}
Jul 28 13:22:12.515889 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.515511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerStarted","Data":"8d10810064778a7f1a63b5db61d0655f7d1336ea98455103bb2b1727e798306d"}
Jul 28 13:22:12.515889 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.515838    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:12.515889 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.515886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:12.517464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.517436    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:12.535594 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:12.535549    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podStartSLOduration=3.529738345 podStartE2EDuration="8.535534241s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.005795896s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:22:12.534070373 +0000 UTC m=+2839.019694968" watchObservedRunningTime="2026-07-28 13:22:12.535534241 +0000 UTC m=+2839.021158871"
Jul 28 13:22:13.520669 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:13.520633    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:18.526157 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:18.526125    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:22:18.526802 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:18.526771    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:28.527444 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:28.527401    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:38.526946 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:38.526908    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:48.527024 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:48.526935    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:22:58.527429 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:22:58.527394    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:23:06.053124 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:06.053085    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:23:06.053593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:06.053306    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" containerID="cri-o://8d10810064778a7f1a63b5db61d0655f7d1336ea98455103bb2b1727e798306d" gracePeriod=30
Jul 28 13:23:06.053593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:06.053341    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kube-rbac-proxy" containerID="cri-o://eb978e310d9c2f8ff84dadf368f98fa5c9013a168bd4a1f72f17309d3ac9e2a9" gracePeriod=30
Jul 28 13:23:06.742955 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:06.742923    2558 generic.go:356] "Generic (PLEG): container finished" podID="f48ea81b-194e-4540-8249-9ecc34357191" containerID="eb978e310d9c2f8ff84dadf368f98fa5c9013a168bd4a1f72f17309d3ac9e2a9" exitCode=2
Jul 28 13:23:06.743123 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:06.742997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerDied","Data":"eb978e310d9c2f8ff84dadf368f98fa5c9013a168bd4a1f72f17309d3ac9e2a9"}
Jul 28 13:23:08.520831 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.520791    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 13:23:08.527518 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.527486    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:23:08.754826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.754795    2558 generic.go:356] "Generic (PLEG): container finished" podID="f48ea81b-194e-4540-8249-9ecc34357191" containerID="8d10810064778a7f1a63b5db61d0655f7d1336ea98455103bb2b1727e798306d" exitCode=0
Jul 28 13:23:08.754982 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.754841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerDied","Data":"8d10810064778a7f1a63b5db61d0655f7d1336ea98455103bb2b1727e798306d"}
Jul 28 13:23:08.803767 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.803714    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:23:08.883244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") pod \"f48ea81b-194e-4540-8249-9ecc34357191\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") "
Jul 28 13:23:08.883408 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883250    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config\") pod \"f48ea81b-194e-4540-8249-9ecc34357191\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") "
Jul 28 13:23:08.883408 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7n4x\" (UniqueName: \"kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x\") pod \"f48ea81b-194e-4540-8249-9ecc34357191\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") "
Jul 28 13:23:08.883408 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883335    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert\") pod \"f48ea81b-194e-4540-8249-9ecc34357191\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") "
Jul 28 13:23:08.883408 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883392    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location\") pod \"f48ea81b-194e-4540-8249-9ecc34357191\" (UID: \"f48ea81b-194e-4540-8249-9ecc34357191\") "
Jul 28 13:23:08.883772 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883743    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f48ea81b-194e-4540-8249-9ecc34357191" (UID: "f48ea81b-194e-4540-8249-9ecc34357191"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:08.883869 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.883846    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f48ea81b-194e-4540-8249-9ecc34357191" (UID: "f48ea81b-194e-4540-8249-9ecc34357191"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:08.885518 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.885498    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f48ea81b-194e-4540-8249-9ecc34357191" (UID: "f48ea81b-194e-4540-8249-9ecc34357191"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:23:08.885887 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.885870    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x" (OuterVolumeSpecName: "kube-api-access-z7n4x") pod "f48ea81b-194e-4540-8249-9ecc34357191" (UID: "f48ea81b-194e-4540-8249-9ecc34357191"). InnerVolumeSpecName "kube-api-access-z7n4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:23:08.893872 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.893844    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f48ea81b-194e-4540-8249-9ecc34357191" (UID: "f48ea81b-194e-4540-8249-9ecc34357191"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:23:08.984069 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.984030    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:23:08.984069 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.984065    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f48ea81b-194e-4540-8249-9ecc34357191-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:23:08.984279 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.984083    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f48ea81b-194e-4540-8249-9ecc34357191-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:23:08.984279 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.984096    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f48ea81b-194e-4540-8249-9ecc34357191-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:23:08.984279 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:08.984107    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7n4x\" (UniqueName: \"kubernetes.io/projected/f48ea81b-194e-4540-8249-9ecc34357191-kube-api-access-z7n4x\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:23:09.760804 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.760773    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"
Jul 28 13:23:09.760804 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.760786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj" event={"ID":"f48ea81b-194e-4540-8249-9ecc34357191","Type":"ContainerDied","Data":"4996458a40ea02416d48fad6d7ca1462e15310965c45c30718ae169eb9ffa79b"}
Jul 28 13:23:09.761299 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.760875    2558 scope.go:117] "RemoveContainer" containerID="eb978e310d9c2f8ff84dadf368f98fa5c9013a168bd4a1f72f17309d3ac9e2a9"
Jul 28 13:23:09.769706 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.769682    2558 scope.go:117] "RemoveContainer" containerID="8d10810064778a7f1a63b5db61d0655f7d1336ea98455103bb2b1727e798306d"
Jul 28 13:23:09.777616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.777596    2558 scope.go:117] "RemoveContainer" containerID="bc463f109be9f917a8ad58608ab4ba09c6c47b6fb4ddf2d307013b4d026b2ca2"
Jul 28 13:23:09.785816 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.785796    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:23:09.792234 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:09.792213    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-dq6zj"]
Jul 28 13:23:10.017222 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:23:10.017142    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48ea81b-194e-4540-8249-9ecc34357191" path="/var/lib/kubelet/pods/f48ea81b-194e-4540-8249-9ecc34357191/volumes"
Jul 28 13:24:28.250510 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250471    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250782    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kube-rbac-proxy"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250793    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kube-rbac-proxy"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250804    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250809    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250826    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="storage-initializer"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250831    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="storage-initializer"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250881    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kserve-container"
Jul 28 13:24:28.250965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.250889    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48ea81b-194e-4540-8249-9ecc34357191" containerName="kube-rbac-proxy"
Jul 28 13:24:28.254255 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.254235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.257266 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.257244    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:24:28.257413 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.257283    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:24:28.257789 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.257768    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:24:28.258318 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.258295    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:24:28.258318 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.258310    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 13:24:28.258552 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.258539    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 13:24:28.258613 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.258564    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:24:28.266698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.266675    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:24:28.319936 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.319894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxzhr\" (UniqueName: \"kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.320132 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.319949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.320132 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.320038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.320132 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.320073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.320273 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.320136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.420884 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.420845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.420892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.420911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.420948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421074 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.421017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxzhr\" (UniqueName: \"kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421289 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:24:28.421083    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:24:28.421289 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:24:28.421170    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls podName:c38159b7-b2b5-4de9-831b-d3993a9cb33a nodeName:}" failed. No retries permitted until 2026-07-28 13:24:28.921148837 +0000 UTC m=+2975.406773407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls") pod "isvc-pmml-predictor-8674795f59-qgh7j" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:24:28.421423 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.421400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421647 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.421624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.421737 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.421634    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.430213 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.430192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxzhr\" (UniqueName: \"kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.925226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.925183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:28.927771 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:28.927714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qgh7j\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:29.164616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:29.164577    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:29.293404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:29.293378    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:24:29.295182 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:24:29.295154    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38159b7_b2b5_4de9_831b_d3993a9cb33a.slice/crio-01cbd6cae88f2efcfe13209ccb9168691b186f936dd21b8a11ff5746ce8f372c WatchSource:0}: Error finding container 01cbd6cae88f2efcfe13209ccb9168691b186f936dd21b8a11ff5746ce8f372c: Status 404 returned error can't find the container with id 01cbd6cae88f2efcfe13209ccb9168691b186f936dd21b8a11ff5746ce8f372c
Jul 28 13:24:30.080524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:30.080483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerStarted","Data":"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"}
Jul 28 13:24:30.080749 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:30.080562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerStarted","Data":"01cbd6cae88f2efcfe13209ccb9168691b186f936dd21b8a11ff5746ce8f372c"}
Jul 28 13:24:34.099663 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:34.099628    2558 generic.go:356] "Generic (PLEG): container finished" podID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerID="20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d" exitCode=0
Jul 28 13:24:34.100135 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:34.099700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerDied","Data":"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"}
Jul 28 13:24:34.100970 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:34.100955    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:24:41.140235 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:41.140194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerStarted","Data":"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"}
Jul 28 13:24:41.140661 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:41.140255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerStarted","Data":"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"}
Jul 28 13:24:41.140661 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:41.140416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:41.166030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:41.165985    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podStartSLOduration=2.390359746 podStartE2EDuration="13.165971055s" totalImagesPullingTime="6.865968911s" totalInitContainerRuntime="3.909642398s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:24:41.164256411 +0000 UTC m=+2987.649881002" watchObservedRunningTime="2026-07-28 13:24:41.165971055 +0000 UTC m=+2987.651595646"
Jul 28 13:24:42.144823 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:42.144790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:42.146191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:42.146160    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:24:43.149342 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:43.149300    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:24:48.153409 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:48.153375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:24:48.153953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:48.153927    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:24:55.109377 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:55.109347    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:24:55.115941 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:55.115920    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:24:58.153974 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:24:58.153929    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:08.153981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:08.153941    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:18.154097 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:18.154058    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:28.153945 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:28.153903    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:38.154637 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:38.154594    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:48.155011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:48.154926    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 13:25:58.154785 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:25:58.154755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:26:09.384378 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:09.384338    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:26:09.384928 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:09.384685    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container" containerID="cri-o://ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42" gracePeriod=30
Jul 28 13:26:09.384928 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:09.384690    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kube-rbac-proxy" containerID="cri-o://49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43" gracePeriod=30
Jul 28 13:26:09.525011 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:09.524975    2558 generic.go:356] "Generic (PLEG): container finished" podID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerID="49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43" exitCode=2
Jul 28 13:26:09.525180 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:09.525047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerDied","Data":"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"}
Jul 28 13:26:13.031551 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.031525    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:26:13.083062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxzhr\" (UniqueName: \"kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr\") pod \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") "
Jul 28 13:26:13.083256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083074    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config\") pod \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") "
Jul 28 13:26:13.083256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083114    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") pod \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") "
Jul 28 13:26:13.083256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083157    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location\") pod \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") "
Jul 28 13:26:13.083256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083220    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert\") pod \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\" (UID: \"c38159b7-b2b5-4de9-831b-d3993a9cb33a\") "
Jul 28 13:26:13.083515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c38159b7-b2b5-4de9-831b-d3993a9cb33a" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:13.083573 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083516    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c38159b7-b2b5-4de9-831b-d3993a9cb33a" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:26:13.083607 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.083593    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c38159b7-b2b5-4de9-831b-d3993a9cb33a" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:13.085208 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.085189    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr" (OuterVolumeSpecName: "kube-api-access-gxzhr") pod "c38159b7-b2b5-4de9-831b-d3993a9cb33a" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a"). InnerVolumeSpecName "kube-api-access-gxzhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:13.085374 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.085353    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c38159b7-b2b5-4de9-831b-d3993a9cb33a" (UID: "c38159b7-b2b5-4de9-831b-d3993a9cb33a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:26:13.183879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.183780    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:26:13.183879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.183814    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:26:13.183879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.183827    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxzhr\" (UniqueName: \"kubernetes.io/projected/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-api-access-gxzhr\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:26:13.183879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.183838    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38159b7-b2b5-4de9-831b-d3993a9cb33a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:26:13.183879 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.183849    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38159b7-b2b5-4de9-831b-d3993a9cb33a-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:26:13.542055 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.541968    2558 generic.go:356] "Generic (PLEG): container finished" podID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerID="ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42" exitCode=0
Jul 28 13:26:13.542055 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.542025    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"
Jul 28 13:26:13.542055 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.542034    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerDied","Data":"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"}
Jul 28 13:26:13.542298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.542080    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j" event={"ID":"c38159b7-b2b5-4de9-831b-d3993a9cb33a","Type":"ContainerDied","Data":"01cbd6cae88f2efcfe13209ccb9168691b186f936dd21b8a11ff5746ce8f372c"}
Jul 28 13:26:13.542298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.542109    2558 scope.go:117] "RemoveContainer" containerID="49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"
Jul 28 13:26:13.553071 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.553045    2558 scope.go:117] "RemoveContainer" containerID="ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"
Jul 28 13:26:13.561244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.561222    2558 scope.go:117] "RemoveContainer" containerID="20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"
Jul 28 13:26:13.564303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.564281    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:26:13.567699 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.567679    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qgh7j"]
Jul 28 13:26:13.569784 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.569767    2558 scope.go:117] "RemoveContainer" containerID="49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"
Jul 28 13:26:13.570081 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:26:13.570060    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43\": container with ID starting with 49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43 not found: ID does not exist" containerID="49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"
Jul 28 13:26:13.570134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.570092    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43"} err="failed to get container status \"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43\": rpc error: code = NotFound desc = could not find container \"49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43\": container with ID starting with 49687a0b0ab22ebbd592dbb43ecd250c1cf8c657b8fdd121124fee69c65fcb43 not found: ID does not exist"
Jul 28 13:26:13.570134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.570110    2558 scope.go:117] "RemoveContainer" containerID="ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"
Jul 28 13:26:13.570360 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:26:13.570346    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42\": container with ID starting with ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42 not found: ID does not exist" containerID="ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"
Jul 28 13:26:13.570406 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.570363    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42"} err="failed to get container status \"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42\": rpc error: code = NotFound desc = could not find container \"ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42\": container with ID starting with ae5276ed79bf1d5f0dd771220b0ca392e46276f952d0710b9160bf53a7d6af42 not found: ID does not exist"
Jul 28 13:26:13.570406 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.570375    2558 scope.go:117] "RemoveContainer" containerID="20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"
Jul 28 13:26:13.570643 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:26:13.570626    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d\": container with ID starting with 20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d not found: ID does not exist" containerID="20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"
Jul 28 13:26:13.570699 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:13.570654    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d"} err="failed to get container status \"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d\": rpc error: code = NotFound desc = could not find container \"20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d\": container with ID starting with 20d760905174746f5d4dd2caeda2f6734688d2061b79e7480651f5651e852c8d not found: ID does not exist"
Jul 28 13:26:14.020957 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.018660    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" path="/var/lib/kubelet/pods/c38159b7-b2b5-4de9-831b-d3993a9cb33a/volumes"
Jul 28 13:26:14.533783 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.533746    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534041    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534050    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534061    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="storage-initializer"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534066    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="storage-initializer"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534078    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kube-rbac-proxy"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534084    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kube-rbac-proxy"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534137    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kserve-container"
Jul 28 13:26:14.534161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.534144    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38159b7-b2b5-4de9-831b-d3993a9cb33a" containerName="kube-rbac-proxy"
Jul 28 13:26:14.537562 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.537542    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.540487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540463    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 13:26:14.540606 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540472    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:26:14.540606 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540507    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:26:14.540754 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540631    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:26:14.540754 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540668    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:26:14.540865 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.540755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:26:14.541692 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.541676    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:26:14.548285 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.548264    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:26:14.594981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.594934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.594981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.594974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.595237 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.595007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbsk\" (UniqueName: \"kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.595237 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.595098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.595237 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.595130    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.695809 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.695768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.695809 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.695813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbsk\" (UniqueName: \"kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696131 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.695884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696131 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.695903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696131 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.695935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696131 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:26:14.696040    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 13:26:14.696131 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:26:14.696117    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls podName:f3416403-edc6-446a-9af5-0f89c18fc234 nodeName:}" failed. No retries permitted until 2026-07-28 13:26:15.196096897 +0000 UTC m=+3081.681721496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-lnf6k" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 13:26:14.696383 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.696314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696497 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.696477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.696551 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.696530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:14.704934 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:14.704913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbsk\" (UniqueName: \"kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:15.200989 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:15.200952    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:15.203397 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:15.203366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lnf6k\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:15.449991 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:15.449951    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:15.580243 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:15.580217    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:26:15.582606 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:26:15.582579    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3416403_edc6_446a_9af5_0f89c18fc234.slice/crio-9d99b511c0c84a154fca4486ff91b4f29b3c1709018982d6534a05f1b021dfb0 WatchSource:0}: Error finding container 9d99b511c0c84a154fca4486ff91b4f29b3c1709018982d6534a05f1b021dfb0: Status 404 returned error can't find the container with id 9d99b511c0c84a154fca4486ff91b4f29b3c1709018982d6534a05f1b021dfb0
Jul 28 13:26:16.560317 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:16.560279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerStarted","Data":"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"}
Jul 28 13:26:16.560523 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:16.560340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerStarted","Data":"9d99b511c0c84a154fca4486ff91b4f29b3c1709018982d6534a05f1b021dfb0"}
Jul 28 13:26:19.573918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:19.573829    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3416403-edc6-446a-9af5-0f89c18fc234" containerID="00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef" exitCode=0
Jul 28 13:26:19.573918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:19.573871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerDied","Data":"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"}
Jul 28 13:26:20.579405 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:20.579371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerStarted","Data":"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"}
Jul 28 13:26:20.579814 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:20.579435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerStarted","Data":"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"}
Jul 28 13:26:20.579814 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:20.579622    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:20.600076 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:20.600025    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podStartSLOduration=2.962008474 podStartE2EDuration="6.600008293s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.637999819s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:20.598011319 +0000 UTC m=+3087.083635910" watchObservedRunningTime="2026-07-28 13:26:20.600008293 +0000 UTC m=+3087.085632966"
Jul 28 13:26:21.583735 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:21.583680    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:21.584841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:21.584814    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:26:22.587791 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:22.587748    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:26:27.593382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:27.593355    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:26:27.593899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:27.593871    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:26:37.594567 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:37.594522    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:26:47.594276 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:47.594237    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:26:57.594515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:26:57.594464    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:27:07.594700 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:07.594660    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:27:17.594035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:17.593946    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:27:27.593958 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:27.593916    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:27:37.594901 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:37.594861    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:27:45.574410 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:45.574378    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:27:45.574997 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:45.574681    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" containerID="cri-o://169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9" gracePeriod=30
Jul 28 13:27:45.574997 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:45.574688    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kube-rbac-proxy" containerID="cri-o://55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf" gracePeriod=30
Jul 28 13:27:45.935462 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:45.935424    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3416403-edc6-446a-9af5-0f89c18fc234" containerID="55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf" exitCode=2
Jul 28 13:27:45.935626 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:45.935492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerDied","Data":"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"}
Jul 28 13:27:47.588869 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:47.588825    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 28 13:27:47.594258 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:47.594230    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 13:27:49.225743 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.225703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:27:49.381054 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config\") pod \"f3416403-edc6-446a-9af5-0f89c18fc234\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") "
Jul 28 13:27:49.381244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381070    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location\") pod \"f3416403-edc6-446a-9af5-0f89c18fc234\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") "
Jul 28 13:27:49.381244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381109    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") pod \"f3416403-edc6-446a-9af5-0f89c18fc234\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") "
Jul 28 13:27:49.381244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381141    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wbsk\" (UniqueName: \"kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk\") pod \"f3416403-edc6-446a-9af5-0f89c18fc234\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") "
Jul 28 13:27:49.381244 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381169    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert\") pod \"f3416403-edc6-446a-9af5-0f89c18fc234\" (UID: \"f3416403-edc6-446a-9af5-0f89c18fc234\") "
Jul 28 13:27:49.381464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3416403-edc6-446a-9af5-0f89c18fc234" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:49.381464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381424    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3416403-edc6-446a-9af5-0f89c18fc234" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:27:49.381644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.381618    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3416403-edc6-446a-9af5-0f89c18fc234" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:49.383384 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.383352    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk" (OuterVolumeSpecName: "kube-api-access-4wbsk") pod "f3416403-edc6-446a-9af5-0f89c18fc234" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234"). InnerVolumeSpecName "kube-api-access-4wbsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:27:49.383384 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.383367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3416403-edc6-446a-9af5-0f89c18fc234" (UID: "f3416403-edc6-446a-9af5-0f89c18fc234"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:27:49.482475 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.482443    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:27:49.482475 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.482471    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3416403-edc6-446a-9af5-0f89c18fc234-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:27:49.482475 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.482482    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3416403-edc6-446a-9af5-0f89c18fc234-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:27:49.482690 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.482492    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbsk\" (UniqueName: \"kubernetes.io/projected/f3416403-edc6-446a-9af5-0f89c18fc234-kube-api-access-4wbsk\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:27:49.482690 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.482500    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3416403-edc6-446a-9af5-0f89c18fc234-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:27:49.953796 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.953760    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3416403-edc6-446a-9af5-0f89c18fc234" containerID="169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9" exitCode=0
Jul 28 13:27:49.953993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.953821    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerDied","Data":"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"}
Jul 28 13:27:49.953993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.953843    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"
Jul 28 13:27:49.953993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.953894    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k" event={"ID":"f3416403-edc6-446a-9af5-0f89c18fc234","Type":"ContainerDied","Data":"9d99b511c0c84a154fca4486ff91b4f29b3c1709018982d6534a05f1b021dfb0"}
Jul 28 13:27:49.953993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.953920    2558 scope.go:117] "RemoveContainer" containerID="55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"
Jul 28 13:27:49.963041 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.963020    2558 scope.go:117] "RemoveContainer" containerID="169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"
Jul 28 13:27:49.971190 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.971174    2558 scope.go:117] "RemoveContainer" containerID="00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"
Jul 28 13:27:49.978995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.978963    2558 scope.go:117] "RemoveContainer" containerID="55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"
Jul 28 13:27:49.979412 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:27:49.979290    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf\": container with ID starting with 55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf not found: ID does not exist" containerID="55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"
Jul 28 13:27:49.979412 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.979330    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf"} err="failed to get container status \"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf\": rpc error: code = NotFound desc = could not find container \"55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf\": container with ID starting with 55c3a857319be75dfc7b8da2589a60663078b74fcdc8892eb2969e4ac8eea0cf not found: ID does not exist"
Jul 28 13:27:49.979412 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.979356    2558 scope.go:117] "RemoveContainer" containerID="169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"
Jul 28 13:27:49.980297 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:27:49.979651    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9\": container with ID starting with 169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9 not found: ID does not exist" containerID="169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"
Jul 28 13:27:49.980297 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.979685    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9"} err="failed to get container status \"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9\": rpc error: code = NotFound desc = could not find container \"169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9\": container with ID starting with 169193f1454c6ea55bffb15272497afccf74ffeb473f775d8996bd2cf00a97a9 not found: ID does not exist"
Jul 28 13:27:49.980297 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.979705    2558 scope.go:117] "RemoveContainer" containerID="00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"
Jul 28 13:27:49.980297 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:27:49.980039    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef\": container with ID starting with 00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef not found: ID does not exist" containerID="00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"
Jul 28 13:27:49.980297 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.980069    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef"} err="failed to get container status \"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef\": rpc error: code = NotFound desc = could not find container \"00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef\": container with ID starting with 00b53bd1c67d8aabaae5b5d75ad9954ba6f958d5dfce308916dcffdae37c09ef not found: ID does not exist"
Jul 28 13:27:49.981539 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.981516    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:27:49.983079 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:49.983058    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lnf6k"]
Jul 28 13:27:50.017178 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.017147    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" path="/var/lib/kubelet/pods/f3416403-edc6-446a-9af5-0f89c18fc234/volumes"
Jul 28 13:27:50.725452 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725414    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725749    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kube-rbac-proxy"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725762    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kube-rbac-proxy"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725780    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="storage-initializer"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725789    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="storage-initializer"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725805    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725812    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725867    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kserve-container"
Jul 28 13:27:50.725943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.725876    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3416403-edc6-446a-9af5-0f89c18fc234" containerName="kube-rbac-proxy"
Jul 28 13:27:50.729409 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.729390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.732270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.732243    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:27:50.732270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.732243    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 13:27:50.732429 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.732243    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:27:50.732600 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.732587    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:27:50.733455 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.733438    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:27:50.733555 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.733459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:27:50.733555 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.733493    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:27:50.739975 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.739954    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:27:50.792839 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.792804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.793036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.792850    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.793036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.792931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.793036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.792962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.793036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.792990    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbnxv\" (UniqueName: \"kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893349 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893349 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893604 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893604 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893604 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbnxv\" (UniqueName: \"kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.893877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.893854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.894186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.894165    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.894186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.894177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.895972 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.895954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:50.902217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:50.902193    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbnxv\" (UniqueName: \"kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:51.040277 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:51.040184    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:51.171403 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:51.171374    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:27:51.173193 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:27:51.173164    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071e0098_6b86_477f_90f0_f09d76836b17.slice/crio-d20e528d15911f451f04cdb456c26d60eae6e3815124ba9f4de266d90a13b303 WatchSource:0}: Error finding container d20e528d15911f451f04cdb456c26d60eae6e3815124ba9f4de266d90a13b303: Status 404 returned error can't find the container with id d20e528d15911f451f04cdb456c26d60eae6e3815124ba9f4de266d90a13b303
Jul 28 13:27:51.964334 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:51.964294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerStarted","Data":"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"}
Jul 28 13:27:51.964695 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:51.964395    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerStarted","Data":"d20e528d15911f451f04cdb456c26d60eae6e3815124ba9f4de266d90a13b303"}
Jul 28 13:27:54.978918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:54.978887    2558 generic.go:356] "Generic (PLEG): container finished" podID="071e0098-6b86-477f-90f0-f09d76836b17" containerID="e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd" exitCode=0
Jul 28 13:27:54.979282 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:54.978961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerDied","Data":"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"}
Jul 28 13:27:55.986149 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:55.986114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerStarted","Data":"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"}
Jul 28 13:27:55.986514 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:55.986183    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerStarted","Data":"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"}
Jul 28 13:27:55.986514 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:55.986376    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:56.008161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:56.008105    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podStartSLOduration=2.367744781 podStartE2EDuration="6.008091007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.640346226s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:27:56.007084974 +0000 UTC m=+3182.492709590" watchObservedRunningTime="2026-07-28 13:27:56.008091007 +0000 UTC m=+3182.493715598"
Jul 28 13:27:56.990542 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:56.990508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:27:56.991981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:56.991952    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:27:57.994561 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:27:57.994519    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:02.998616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:02.998586    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:28:02.999133 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:02.999107    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:12.999958 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:12.999915    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:22.999636 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:22.999597    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:32.998991 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:32.998953    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:42.999252 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:42.999212    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:28:52.999936 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:28:52.999893    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:29:02.999605 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:02.999561    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:29:12.999863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:12.999833    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:29:21.561926 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:21.561893    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:29:21.562501 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:21.562131    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" containerID="cri-o://1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b" gracePeriod=30
Jul 28 13:29:21.562501 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:21.562167    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kube-rbac-proxy" containerID="cri-o://e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69" gracePeriod=30
Jul 28 13:29:22.326006 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:22.325974    2558 generic.go:356] "Generic (PLEG): container finished" podID="071e0098-6b86-477f-90f0-f09d76836b17" containerID="e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69" exitCode=2
Jul 28 13:29:22.326006 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:22.326011    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerDied","Data":"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"}
Jul 28 13:29:22.995642 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:22.995595    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 28 13:29:22.999978 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:22.999942    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 13:29:25.209671 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.209647    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:29:25.321125 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321035    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls\") pod \"071e0098-6b86-477f-90f0-f09d76836b17\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") "
Jul 28 13:29:25.321125 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321076    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location\") pod \"071e0098-6b86-477f-90f0-f09d76836b17\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") "
Jul 28 13:29:25.321125 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config\") pod \"071e0098-6b86-477f-90f0-f09d76836b17\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") "
Jul 28 13:29:25.321361 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321138    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbnxv\" (UniqueName: \"kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv\") pod \"071e0098-6b86-477f-90f0-f09d76836b17\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") "
Jul 28 13:29:25.321361 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321226    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert\") pod \"071e0098-6b86-477f-90f0-f09d76836b17\" (UID: \"071e0098-6b86-477f-90f0-f09d76836b17\") "
Jul 28 13:29:25.321501 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "071e0098-6b86-477f-90f0-f09d76836b17" (UID: "071e0098-6b86-477f-90f0-f09d76836b17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:29:25.321808 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321594    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "071e0098-6b86-477f-90f0-f09d76836b17" (UID: "071e0098-6b86-477f-90f0-f09d76836b17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:29:25.321808 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.321650    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "071e0098-6b86-477f-90f0-f09d76836b17" (UID: "071e0098-6b86-477f-90f0-f09d76836b17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:29:25.323396 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.323367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv" (OuterVolumeSpecName: "kube-api-access-fbnxv") pod "071e0098-6b86-477f-90f0-f09d76836b17" (UID: "071e0098-6b86-477f-90f0-f09d76836b17"). InnerVolumeSpecName "kube-api-access-fbnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:29:25.323505 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.323375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "071e0098-6b86-477f-90f0-f09d76836b17" (UID: "071e0098-6b86-477f-90f0-f09d76836b17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:29:25.340571 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.340544    2558 generic.go:356] "Generic (PLEG): container finished" podID="071e0098-6b86-477f-90f0-f09d76836b17" containerID="1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b" exitCode=0
Jul 28 13:29:25.340753 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.340622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"
Jul 28 13:29:25.340753 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.340619    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerDied","Data":"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"}
Jul 28 13:29:25.340877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.340771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm" event={"ID":"071e0098-6b86-477f-90f0-f09d76836b17","Type":"ContainerDied","Data":"d20e528d15911f451f04cdb456c26d60eae6e3815124ba9f4de266d90a13b303"}
Jul 28 13:29:25.340877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.340801    2558 scope.go:117] "RemoveContainer" containerID="e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"
Jul 28 13:29:25.348854 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.348835    2558 scope.go:117] "RemoveContainer" containerID="1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"
Jul 28 13:29:25.356017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.356002    2558 scope.go:117] "RemoveContainer" containerID="e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"
Jul 28 13:29:25.362899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.362881    2558 scope.go:117] "RemoveContainer" containerID="e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"
Jul 28 13:29:25.363145 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:29:25.363126    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69\": container with ID starting with e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69 not found: ID does not exist" containerID="e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"
Jul 28 13:29:25.363190 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.363153    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69"} err="failed to get container status \"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69\": rpc error: code = NotFound desc = could not find container \"e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69\": container with ID starting with e39d42f7b2665968f1946df9d84635175fbb3834f5ff806055f5d75491dbfc69 not found: ID does not exist"
Jul 28 13:29:25.363190 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.363169    2558 scope.go:117] "RemoveContainer" containerID="1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"
Jul 28 13:29:25.363368 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:29:25.363351    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b\": container with ID starting with 1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b not found: ID does not exist" containerID="1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"
Jul 28 13:29:25.363407 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.363373    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b"} err="failed to get container status \"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b\": rpc error: code = NotFound desc = could not find container \"1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b\": container with ID starting with 1fad7866fe048134e4c70150b7d3cf13f158ea88188f220be01d0532efa3b99b not found: ID does not exist"
Jul 28 13:29:25.363407 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.363386    2558 scope.go:117] "RemoveContainer" containerID="e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"
Jul 28 13:29:25.363569 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:29:25.363553    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd\": container with ID starting with e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd not found: ID does not exist" containerID="e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"
Jul 28 13:29:25.363609 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.363572    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd"} err="failed to get container status \"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd\": rpc error: code = NotFound desc = could not find container \"e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd\": container with ID starting with e63ea10d68f40f86de8d3124d39a5288610a2af9fb682ca6a4c9addb3d2675dd not found: ID does not exist"
Jul 28 13:29:25.366581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.366561    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:29:25.371454 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.371431    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9wfqm"]
Jul 28 13:29:25.422547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.422514    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:29:25.422547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.422542    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/071e0098-6b86-477f-90f0-f09d76836b17-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:29:25.422547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.422551    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071e0098-6b86-477f-90f0-f09d76836b17-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:29:25.422793 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.422561    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/071e0098-6b86-477f-90f0-f09d76836b17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:29:25.422793 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:25.422573    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbnxv\" (UniqueName: \"kubernetes.io/projected/071e0098-6b86-477f-90f0-f09d76836b17-kube-api-access-fbnxv\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:29:26.017053 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:26.017019    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="071e0098-6b86-477f-90f0-f09d76836b17" path="/var/lib/kubelet/pods/071e0098-6b86-477f-90f0-f09d76836b17/volumes"
Jul 28 13:29:55.132322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:55.132210    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:29:55.139218 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:29:55.139200    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:31:11.985779 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.985739    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986063    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kube-rbac-proxy"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986075    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kube-rbac-proxy"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986086    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="storage-initializer"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986092    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="storage-initializer"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986103    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986169    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kube-rbac-proxy"
Jul 28 13:31:11.986191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.986179    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="071e0098-6b86-477f-90f0-f09d76836b17" containerName="kserve-container"
Jul 28 13:31:11.989033 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.989017    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:11.991844 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.991822    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:31:11.992046 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.992026    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a95d50-predictor-serving-cert\""
Jul 28 13:31:11.992128 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.992056    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:31:11.992179 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.992129    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a95d50\""
Jul 28 13:31:11.992179 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.992135    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:31:11.993062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.993046    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a95d50-kube-rbac-proxy-sar-config\""
Jul 28 13:31:11.993233 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.993219    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a95d50-dockercfg-zt6xb\""
Jul 28 13:31:11.998702 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:11.998682    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:12.169169 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.169127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.169169 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.169172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.169395 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.169199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.169395 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.169302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.169395 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.169336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q62h\" (UniqueName: \"kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.270782 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.270664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.270782 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.270708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q62h\" (UniqueName: \"kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.270995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.270784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.270995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.270822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.270995 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.270856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.271241 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.271210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.271477 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.271457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.271573 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.271551    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.273359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.273341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.279684 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.279662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q62h\" (UniqueName: \"kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h\") pod \"isvc-secondary-a95d50-predictor-596667c754-gcrq2\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") " pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.299557 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.299522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:12.431826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.431801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:12.433758 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:31:12.433709    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3f98a5_d699_4570_9819_dfe1d170e773.slice/crio-ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9 WatchSource:0}: Error finding container ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9: Status 404 returned error can't find the container with id ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9
Jul 28 13:31:12.770327 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.770288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerStarted","Data":"0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"}
Jul 28 13:31:12.770533 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:12.770356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerStarted","Data":"ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9"}
Jul 28 13:31:17.791502 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:17.791424    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/0.log"
Jul 28 13:31:17.791502 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:17.791464    2558 generic.go:356] "Generic (PLEG): container finished" podID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerID="0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495" exitCode=1
Jul 28 13:31:17.791932 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:17.791511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerDied","Data":"0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"}
Jul 28 13:31:18.797339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:18.797313    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/0.log"
Jul 28 13:31:18.797761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:18.797382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerStarted","Data":"290480cdad642354d0e929593a9d260c1130b62d766046c87dffa7f0f63f92d2"}
Jul 28 13:31:22.816761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.816714    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/1.log"
Jul 28 13:31:22.817183 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.817158    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/0.log"
Jul 28 13:31:22.817267 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.817198    2558 generic.go:356] "Generic (PLEG): container finished" podID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerID="290480cdad642354d0e929593a9d260c1130b62d766046c87dffa7f0f63f92d2" exitCode=1
Jul 28 13:31:22.817267 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.817239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerDied","Data":"290480cdad642354d0e929593a9d260c1130b62d766046c87dffa7f0f63f92d2"}
Jul 28 13:31:22.817372 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.817314    2558 scope.go:117] "RemoveContainer" containerID="0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"
Jul 28 13:31:22.817735 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.817698    2558 scope.go:117] "RemoveContainer" containerID="0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"
Jul 28 13:31:22.827951 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:22.827925    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a95d50-predictor-596667c754-gcrq2_kserve-ci-e2e-test(ce3f98a5-d699-4570-9819-dfe1d170e773)\"" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773"
Jul 28 13:31:22.828059 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:22.828021    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a95d50-predictor-596667c754-gcrq2_kserve-ci-e2e-test_ce3f98a5-d699-4570-9819-dfe1d170e773_0 in pod sandbox ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9 from index: no such id: '0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495'" containerID="0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"
Jul 28 13:31:22.828120 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:22.828058    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a95d50-predictor-596667c754-gcrq2_kserve-ci-e2e-test_ce3f98a5-d699-4570-9819-dfe1d170e773_0 in pod sandbox ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9 from index: no such id: '0f9a16ad283b51fefbb4e90cbcc6aec4f577dc3110f254e1e48ef75dd9989495'"
Jul 28 13:31:23.822975 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:23.822949    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/1.log"
Jul 28 13:31:30.104901 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.104857    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:30.236841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.236814    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/1.log"
Jul 28 13:31:30.240245 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.240230    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:30.320866 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.320815    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config\") pod \"ce3f98a5-d699-4570-9819-dfe1d170e773\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") "
Jul 28 13:31:30.321051 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.320889    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q62h\" (UniqueName: \"kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h\") pod \"ce3f98a5-d699-4570-9819-dfe1d170e773\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") "
Jul 28 13:31:30.321051 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.320918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert\") pod \"ce3f98a5-d699-4570-9819-dfe1d170e773\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") "
Jul 28 13:31:30.321051 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.320981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location\") pod \"ce3f98a5-d699-4570-9819-dfe1d170e773\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") "
Jul 28 13:31:30.321051 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.321010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls\") pod \"ce3f98a5-d699-4570-9819-dfe1d170e773\" (UID: \"ce3f98a5-d699-4570-9819-dfe1d170e773\") "
Jul 28 13:31:30.321322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.321293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce3f98a5-d699-4570-9819-dfe1d170e773" (UID: "ce3f98a5-d699-4570-9819-dfe1d170e773"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:30.321322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.321303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce3f98a5-d699-4570-9819-dfe1d170e773" (UID: "ce3f98a5-d699-4570-9819-dfe1d170e773"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:31:30.321412 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.321377    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce3f98a5-d699-4570-9819-dfe1d170e773" (UID: "ce3f98a5-d699-4570-9819-dfe1d170e773"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:30.323214 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.323188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h" (OuterVolumeSpecName: "kube-api-access-8q62h") pod "ce3f98a5-d699-4570-9819-dfe1d170e773" (UID: "ce3f98a5-d699-4570-9819-dfe1d170e773"). InnerVolumeSpecName "kube-api-access-8q62h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:31:30.323323 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.323240    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce3f98a5-d699-4570-9819-dfe1d170e773" (UID: "ce3f98a5-d699-4570-9819-dfe1d170e773"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:31:30.421842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.421745    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q62h\" (UniqueName: \"kubernetes.io/projected/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-api-access-8q62h\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:30.421842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.421790    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:30.421842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.421799    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3f98a5-d699-4570-9819-dfe1d170e773-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:30.421842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.421809    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3f98a5-d699-4570-9819-dfe1d170e773-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:30.421842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.421819    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3f98a5-d699-4570-9819-dfe1d170e773-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:30.850658 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.850630    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a95d50-predictor-596667c754-gcrq2_ce3f98a5-d699-4570-9819-dfe1d170e773/storage-initializer/1.log"
Jul 28 13:31:30.850838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.850699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2" event={"ID":"ce3f98a5-d699-4570-9819-dfe1d170e773","Type":"ContainerDied","Data":"ef13ed5caad693976cb24ba77c8c58d96386876f909acd54ad805998a954d6d9"}
Jul 28 13:31:30.850838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.850742    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"
Jul 28 13:31:30.850838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.850778    2558 scope.go:117] "RemoveContainer" containerID="290480cdad642354d0e929593a9d260c1130b62d766046c87dffa7f0f63f92d2"
Jul 28 13:31:30.886295 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.886262    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:30.889707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:30.889678    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a95d50-predictor-596667c754-gcrq2"]
Jul 28 13:31:32.017447 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:32.017403    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" path="/var/lib/kubelet/pods/ce3f98a5-d699-4570-9819-dfe1d170e773/volumes"
Jul 28 13:31:35.256803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.256770    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:35.257217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257181    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.257217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257196    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.257217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257211    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.257342 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257220    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.257342 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257306    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.257342 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.257316    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3f98a5-d699-4570-9819-dfe1d170e773" containerName="storage-initializer"
Jul 28 13:31:35.262133 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.262111    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.264855 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.264830    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e48a5b\""
Jul 28 13:31:35.264989 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.264862    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e48a5b-predictor-serving-cert\""
Jul 28 13:31:35.265174 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.265153    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:31:35.265237 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.265161    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e48a5b-kube-rbac-proxy-sar-config\""
Jul 28 13:31:35.266107 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.266088    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:31:35.266212 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.266115    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e48a5b-dockercfg-9c72p\""
Jul 28 13:31:35.266212 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.266119    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:31:35.270680 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.270659    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:35.359643 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.359604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.359643 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.359638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xknp\" (UniqueName: \"kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.359921 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.359678    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.359921 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.359702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.359921 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.359732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.460998 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.460957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.460998 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.460993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xknp\" (UniqueName: \"kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461058    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461250 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:35.461239    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-serving-cert: secret "isvc-init-fail-e48a5b-predictor-serving-cert" not found
Jul 28 13:31:35.461460 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:35.461296    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls podName:4f3e0da3-5756-438a-92dd-10d94397aabf nodeName:}" failed. No retries permitted until 2026-07-28 13:31:35.961277482 +0000 UTC m=+3402.446902054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls") pod "isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf") : secret "isvc-init-fail-e48a5b-predictor-serving-cert" not found
Jul 28 13:31:35.461523 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461664 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.461740 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.461708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.470062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.470040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xknp\" (UniqueName: \"kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.964806 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.964767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:35.967281 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:35.967260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") pod \"isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") " pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:36.172700 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:36.172664    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:36.305426 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:36.305397    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:36.307897 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:31:36.307863    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3e0da3_5756_438a_92dd_10d94397aabf.slice/crio-99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f WatchSource:0}: Error finding container 99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f: Status 404 returned error can't find the container with id 99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f
Jul 28 13:31:36.885502 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:36.885461    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerStarted","Data":"2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"}
Jul 28 13:31:36.885705 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:36.885571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerStarted","Data":"99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f"}
Jul 28 13:31:41.907239 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:41.907212    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/0.log"
Jul 28 13:31:41.907612 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:41.907250    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerID="2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c" exitCode=1
Jul 28 13:31:41.907612 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:41.907318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerDied","Data":"2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"}
Jul 28 13:31:42.913194 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:42.913162    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/0.log"
Jul 28 13:31:42.913586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:42.913228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerStarted","Data":"87ececb48a93f6189d42f50e3c0a0a567a53e04150cdada5e82fd6cedffcc2fa"}
Jul 28 13:31:43.918505 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.918478    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/1.log"
Jul 28 13:31:43.918913 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.918848    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/0.log"
Jul 28 13:31:43.918913 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.918882    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerID="87ececb48a93f6189d42f50e3c0a0a567a53e04150cdada5e82fd6cedffcc2fa" exitCode=1
Jul 28 13:31:43.918993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.918953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerDied","Data":"87ececb48a93f6189d42f50e3c0a0a567a53e04150cdada5e82fd6cedffcc2fa"}
Jul 28 13:31:43.919039 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.919017    2558 scope.go:117] "RemoveContainer" containerID="2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"
Jul 28 13:31:43.919373 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.919355    2558 scope.go:117] "RemoveContainer" containerID="2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"
Jul 28 13:31:43.929932 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:43.929910    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_kserve-ci-e2e-test(4f3e0da3-5756-438a-92dd-10d94397aabf)\"" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf"
Jul 28 13:31:43.930193 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:31:43.930164    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_kserve-ci-e2e-test_4f3e0da3-5756-438a-92dd-10d94397aabf_0 in pod sandbox 99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f from index: no such id: '2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c'" containerID="2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"
Jul 28 13:31:43.930310 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:43.930195    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_kserve-ci-e2e-test_4f3e0da3-5756-438a-92dd-10d94397aabf_0 in pod sandbox 99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f from index: no such id: '2668892a34a76dff36c8fac41b0ab396011a32a0a594ad280504e23cf4d8885c'"
Jul 28 13:31:44.924868 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:44.924837    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/1.log"
Jul 28 13:31:45.277323 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.277242    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:45.409482 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.409462    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/1.log"
Jul 28 13:31:45.412952 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.412935    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:45.540326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540201    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") pod \"4f3e0da3-5756-438a-92dd-10d94397aabf\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") "
Jul 28 13:31:45.540326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location\") pod \"4f3e0da3-5756-438a-92dd-10d94397aabf\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") "
Jul 28 13:31:45.540326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xknp\" (UniqueName: \"kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp\") pod \"4f3e0da3-5756-438a-92dd-10d94397aabf\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") "
Jul 28 13:31:45.540326 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config\") pod \"4f3e0da3-5756-438a-92dd-10d94397aabf\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") "
Jul 28 13:31:45.540642 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540345    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert\") pod \"4f3e0da3-5756-438a-92dd-10d94397aabf\" (UID: \"4f3e0da3-5756-438a-92dd-10d94397aabf\") "
Jul 28 13:31:45.540713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f3e0da3-5756-438a-92dd-10d94397aabf" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:31:45.540800 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540758    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f3e0da3-5756-438a-92dd-10d94397aabf" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:45.540800 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.540755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f3e0da3-5756-438a-92dd-10d94397aabf" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:45.542565 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.542543    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp" (OuterVolumeSpecName: "kube-api-access-9xknp") pod "4f3e0da3-5756-438a-92dd-10d94397aabf" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf"). InnerVolumeSpecName "kube-api-access-9xknp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:31:45.542634 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.542596    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f3e0da3-5756-438a-92dd-10d94397aabf" (UID: "4f3e0da3-5756-438a-92dd-10d94397aabf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:31:45.641761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.641701    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:45.641761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.641754    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f3e0da3-5756-438a-92dd-10d94397aabf-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:45.641761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.641763    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f3e0da3-5756-438a-92dd-10d94397aabf-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:45.641761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.641772    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xknp\" (UniqueName: \"kubernetes.io/projected/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-api-access-9xknp\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:45.642030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.641780    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f3e0da3-5756-438a-92dd-10d94397aabf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:31:45.929850 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.929823    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r_4f3e0da3-5756-438a-92dd-10d94397aabf/storage-initializer/1.log"
Jul 28 13:31:45.930327 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.929908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r" event={"ID":"4f3e0da3-5756-438a-92dd-10d94397aabf","Type":"ContainerDied","Data":"99fbd56b60687f8f0d73e205e4b363d101d12b51672b7c28f667791700eeaa1f"}
Jul 28 13:31:45.930327 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.929933    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"
Jul 28 13:31:45.930327 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.929965    2558 scope.go:117] "RemoveContainer" containerID="87ececb48a93f6189d42f50e3c0a0a567a53e04150cdada5e82fd6cedffcc2fa"
Jul 28 13:31:45.967776 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.967735    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:45.972295 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:45.972266    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e48a5b-predictor-78658c844b-8kk9r"]
Jul 28 13:31:46.017347 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:46.017314    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" path="/var/lib/kubelet/pods/4f3e0da3-5756-438a-92dd-10d94397aabf/volumes"
Jul 28 13:31:50.509559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.509527    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:31:50.509937 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.509877    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.509937 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.509890    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.510023 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.509968    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.510023 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.509977    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.510087 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.510035    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.510087 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.510042    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f3e0da3-5756-438a-92dd-10d94397aabf" containerName="storage-initializer"
Jul 28 13:31:50.514489 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.514470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.517298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.517092    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:31:50.517298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.517158    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:31:50.517298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.517167    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:31:50.517298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.517181    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:31:50.517298 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.517193    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:31:50.518231 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.518218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 13:31:50.518303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.518288    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 13:31:50.523146 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.523117    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:31:50.578579 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.578537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.578787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.578626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.578787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.578655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.578787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.578687    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcwb\" (UniqueName: \"kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.578908 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.578792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680101 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcwb\" (UniqueName: \"kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680101 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680343 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680129    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680343 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680343 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680514 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680871 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.680968 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.680849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.682700 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.682683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.689009 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.688977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcwb\" (UniqueName: \"kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z9479\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.826836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.826744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:31:50.959999 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:50.959968    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:31:50.962891 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:31:50.962864    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92adfedd_8573_441f_9b2e_62a88fb8900e.slice/crio-c98d32686f3de75b4011a98e58f89a7c0b01856bfb72f5c079e1789ae6e9a508 WatchSource:0}: Error finding container c98d32686f3de75b4011a98e58f89a7c0b01856bfb72f5c079e1789ae6e9a508: Status 404 returned error can't find the container with id c98d32686f3de75b4011a98e58f89a7c0b01856bfb72f5c079e1789ae6e9a508
Jul 28 13:31:51.957549 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:51.957509    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerStarted","Data":"f9d2956219c03eea8c580cccfd1f65a3748c9ef46fd7b8c72edada80a3debf92"}
Jul 28 13:31:51.957954 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:51.957572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerStarted","Data":"c98d32686f3de75b4011a98e58f89a7c0b01856bfb72f5c079e1789ae6e9a508"}
Jul 28 13:31:54.970899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:54.970800    2558 generic.go:356] "Generic (PLEG): container finished" podID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerID="f9d2956219c03eea8c580cccfd1f65a3748c9ef46fd7b8c72edada80a3debf92" exitCode=0
Jul 28 13:31:54.970899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:54.970873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerDied","Data":"f9d2956219c03eea8c580cccfd1f65a3748c9ef46fd7b8c72edada80a3debf92"}
Jul 28 13:31:54.972145 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:31:54.972130    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:32:18.090032 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:18.089995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerStarted","Data":"67159283b1274e832e7ede6023ed5ef00df326a31ddab7daf80648988a8da561"}
Jul 28 13:32:19.095928 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:19.095892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerStarted","Data":"e682149de0523774ba989654f60214388ede831f544e5ab2f0c5513cac3b6adc"}
Jul 28 13:32:19.096336 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:19.095988    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:32:19.115339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:19.115283    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podStartSLOduration=2.579251477 podStartE2EDuration="29.115269434s" totalImagesPullingTime="22.891761882s" totalInitContainerRuntime="3.644256075s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:32:19.114079171 +0000 UTC m=+3445.599703761" watchObservedRunningTime="2026-07-28 13:32:19.115269434 +0000 UTC m=+3445.600894025"
Jul 28 13:32:20.100541 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:20.100506    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:32:20.101888 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:20.101861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:32:21.104960 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:21.104914    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:32:26.110316 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:26.110283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:32:26.110945 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:26.110917    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:32:36.111868 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:36.111822    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:32:46.111042 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:46.110997    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:32:56.111105 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:32:56.111064    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:33:06.111052 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:06.111013    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:33:16.111250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:16.111159    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:33:26.111005 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:26.110964    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 13:33:29.013531 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:29.013496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:33:30.685257 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:30.685218    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:33:30.685681 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:30.685470    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container" containerID="cri-o://67159283b1274e832e7ede6023ed5ef00df326a31ddab7daf80648988a8da561" gracePeriod=30
Jul 28 13:33:30.685681 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:30.685488    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kube-rbac-proxy" containerID="cri-o://e682149de0523774ba989654f60214388ede831f544e5ab2f0c5513cac3b6adc" gracePeriod=30
Jul 28 13:33:31.105951 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:31.105905    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 13:33:31.398622 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:31.398531    2558 generic.go:356] "Generic (PLEG): container finished" podID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerID="e682149de0523774ba989654f60214388ede831f544e5ab2f0c5513cac3b6adc" exitCode=2
Jul 28 13:33:31.398622 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:31.398569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerDied","Data":"e682149de0523774ba989654f60214388ede831f544e5ab2f0c5513cac3b6adc"}
Jul 28 13:33:35.417029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.416997    2558 generic.go:356] "Generic (PLEG): container finished" podID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerID="67159283b1274e832e7ede6023ed5ef00df326a31ddab7daf80648988a8da561" exitCode=0
Jul 28 13:33:35.417477 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.417080    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerDied","Data":"67159283b1274e832e7ede6023ed5ef00df326a31ddab7daf80648988a8da561"}
Jul 28 13:33:35.446807 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.446785    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:33:35.562828 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.562709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls\") pod \"92adfedd-8573-441f-9b2e-62a88fb8900e\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") "
Jul 28 13:33:35.562828 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.562790    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert\") pod \"92adfedd-8573-441f-9b2e-62a88fb8900e\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") "
Jul 28 13:33:35.562828 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.562817    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config\") pod \"92adfedd-8573-441f-9b2e-62a88fb8900e\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") "
Jul 28 13:33:35.563056 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.562848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbcwb\" (UniqueName: \"kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb\") pod \"92adfedd-8573-441f-9b2e-62a88fb8900e\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") "
Jul 28 13:33:35.563056 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.562914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location\") pod \"92adfedd-8573-441f-9b2e-62a88fb8900e\" (UID: \"92adfedd-8573-441f-9b2e-62a88fb8900e\") "
Jul 28 13:33:35.563204 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.563177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92adfedd-8573-441f-9b2e-62a88fb8900e" (UID: "92adfedd-8573-441f-9b2e-62a88fb8900e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:33:35.563307 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.563284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92adfedd-8573-441f-9b2e-62a88fb8900e" (UID: "92adfedd-8573-441f-9b2e-62a88fb8900e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:33:35.563374 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.563324    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92adfedd-8573-441f-9b2e-62a88fb8900e" (UID: "92adfedd-8573-441f-9b2e-62a88fb8900e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:33:35.565107 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.565084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92adfedd-8573-441f-9b2e-62a88fb8900e" (UID: "92adfedd-8573-441f-9b2e-62a88fb8900e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:33:35.565207 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.565104    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb" (OuterVolumeSpecName: "kube-api-access-nbcwb") pod "92adfedd-8573-441f-9b2e-62a88fb8900e" (UID: "92adfedd-8573-441f-9b2e-62a88fb8900e"). InnerVolumeSpecName "kube-api-access-nbcwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:33:35.663692 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.663658    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adfedd-8573-441f-9b2e-62a88fb8900e-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:33:35.663692 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.663687    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:33:35.663692 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.663697    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:33:35.663941 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.663706    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcwb\" (UniqueName: \"kubernetes.io/projected/92adfedd-8573-441f-9b2e-62a88fb8900e-kube-api-access-nbcwb\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:33:35.663941 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:35.663715    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adfedd-8573-441f-9b2e-62a88fb8900e-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:33:36.422713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.422681    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"
Jul 28 13:33:36.422713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.422699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479" event={"ID":"92adfedd-8573-441f-9b2e-62a88fb8900e","Type":"ContainerDied","Data":"c98d32686f3de75b4011a98e58f89a7c0b01856bfb72f5c079e1789ae6e9a508"}
Jul 28 13:33:36.423214 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.422791    2558 scope.go:117] "RemoveContainer" containerID="e682149de0523774ba989654f60214388ede831f544e5ab2f0c5513cac3b6adc"
Jul 28 13:33:36.431002 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.430983    2558 scope.go:117] "RemoveContainer" containerID="67159283b1274e832e7ede6023ed5ef00df326a31ddab7daf80648988a8da561"
Jul 28 13:33:36.438409 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.438385    2558 scope.go:117] "RemoveContainer" containerID="f9d2956219c03eea8c580cccfd1f65a3748c9ef46fd7b8c72edada80a3debf92"
Jul 28 13:33:36.441884 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.441862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:33:36.444644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:36.444622    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z9479"]
Jul 28 13:33:38.017023 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:33:38.016984    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" path="/var/lib/kubelet/pods/92adfedd-8573-441f-9b2e-62a88fb8900e/volumes"
Jul 28 13:34:55.153148 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:34:55.153043    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:34:55.163125 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:34:55.163106    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:35:31.208865 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.208828    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209125    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209136    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209148    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="storage-initializer"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209154    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="storage-initializer"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209169    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kube-rbac-proxy"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209175    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kube-rbac-proxy"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209218    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kserve-container"
Jul 28 13:35:31.209268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.209227    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="92adfedd-8573-441f-9b2e-62a88fb8900e" containerName="kube-rbac-proxy"
Jul 28 13:35:31.211755 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.211738    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.214783 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.214754    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:35:31.215940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.215913    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 13:35:31.216085 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.215974    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:35:31.216085 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.215995    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 13:35:31.216085 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.216022    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:35:31.216085 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.216026    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:35:31.216322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.216191    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:35:31.222635 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.222611    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:35:31.283815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.283783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.283815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.283821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.284062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.283885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.284062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.283923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rt4f\" (UniqueName: \"kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.284062 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.283975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385249 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385517 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385276    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385517 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385517 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385322    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385517 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rt4f\" (UniqueName: \"kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.385823 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.385798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.386055 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.386031    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.386124 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.386036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.387863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.387847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.394275 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.394252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rt4f\" (UniqueName: \"kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.522359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.522271    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:31.655044 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.655017    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:35:31.657150 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:35:31.657123    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47744136_9332_4dc9_955a_7f25216a88fd.slice/crio-5d5ad3b45e0f9ba17b361b3bd1acfcc299c1b8a799eb4d07dbec25610855c245 WatchSource:0}: Error finding container 5d5ad3b45e0f9ba17b361b3bd1acfcc299c1b8a799eb4d07dbec25610855c245: Status 404 returned error can't find the container with id 5d5ad3b45e0f9ba17b361b3bd1acfcc299c1b8a799eb4d07dbec25610855c245
Jul 28 13:35:31.891514 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.891473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerStarted","Data":"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"}
Jul 28 13:35:31.891738 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:31.891575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerStarted","Data":"5d5ad3b45e0f9ba17b361b3bd1acfcc299c1b8a799eb4d07dbec25610855c245"}
Jul 28 13:35:35.908619 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:35.908585    2558 generic.go:356] "Generic (PLEG): container finished" podID="47744136-9332-4dc9-955a-7f25216a88fd" containerID="86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de" exitCode=0
Jul 28 13:35:35.909018 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:35.908648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerDied","Data":"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"}
Jul 28 13:35:36.914836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:36.914797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerStarted","Data":"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"}
Jul 28 13:35:36.915210 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:36.914867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerStarted","Data":"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"}
Jul 28 13:35:36.915210 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:36.915051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:36.936274 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:36.936216    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podStartSLOduration=2.470095834 podStartE2EDuration="5.936201303s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.466105469s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:35:36.934328753 +0000 UTC m=+3643.419953369" watchObservedRunningTime="2026-07-28 13:35:36.936201303 +0000 UTC m=+3643.421825895"
Jul 28 13:35:37.918641 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:37.918611    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:37.920161 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:37.920131    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:35:38.927791 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:38.927742    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:35:43.932684 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:43.932653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:35:43.933165 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:43.933135    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:35:53.933899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:35:53.933849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:03.933580 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:03.933538    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:13.933784 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:13.933741    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:23.933815 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:23.933773    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:33.933398 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:33.933358    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:43.933541 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:43.933500    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:36:53.933771 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:36:53.933739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:37:01.298268 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:01.298235    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:37:01.298668 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:01.298491    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" containerID="cri-o://dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c" gracePeriod=30
Jul 28 13:37:01.298668 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:01.298567    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kube-rbac-proxy" containerID="cri-o://bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad" gracePeriod=30
Jul 28 13:37:02.265325 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:02.265289    2558 generic.go:356] "Generic (PLEG): container finished" podID="47744136-9332-4dc9-955a-7f25216a88fd" containerID="bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad" exitCode=2
Jul 28 13:37:02.265495 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:02.265336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerDied","Data":"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"}
Jul 28 13:37:03.928452 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:03.928398    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 13:37:03.933877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:03.933844    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 13:37:06.461012 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.460974    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:37:06.463641 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.463625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.466415 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.466392    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 13:37:06.466555 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.466457    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:37:06.473818 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.473794    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:37:06.561211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.561172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.561211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.561215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.561428 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.561296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.561428 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.561321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.561428 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.561376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.661998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662206 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662206 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662206 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662206 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662414 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:37:06.662270    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 13:37:06.662414 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:37:06.662375    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls podName:9f0deb03-2f41-4d27-8986-907042cef819 nodeName:}" failed. No retries permitted until 2026-07-28 13:37:07.162352873 +0000 UTC m=+3733.647977458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" (UID: "9f0deb03-2f41-4d27-8986-907042cef819") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 13:37:06.662584 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662566    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662872 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.662916 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.662851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.671025 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.671001    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:06.844601 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.844574    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:37:06.964274 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964242    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config\") pod \"47744136-9332-4dc9-955a-7f25216a88fd\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") "
Jul 28 13:37:06.964274 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert\") pod \"47744136-9332-4dc9-955a-7f25216a88fd\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") "
Jul 28 13:37:06.964540 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964322    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location\") pod \"47744136-9332-4dc9-955a-7f25216a88fd\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") "
Jul 28 13:37:06.964540 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964427    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls\") pod \"47744136-9332-4dc9-955a-7f25216a88fd\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") "
Jul 28 13:37:06.964540 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rt4f\" (UniqueName: \"kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f\") pod \"47744136-9332-4dc9-955a-7f25216a88fd\" (UID: \"47744136-9332-4dc9-955a-7f25216a88fd\") "
Jul 28 13:37:06.964758 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964695    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47744136-9332-4dc9-955a-7f25216a88fd" (UID: "47744136-9332-4dc9-955a-7f25216a88fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:37:06.964758 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47744136-9332-4dc9-955a-7f25216a88fd" (UID: "47744136-9332-4dc9-955a-7f25216a88fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:37:06.964758 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.964715    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47744136-9332-4dc9-955a-7f25216a88fd" (UID: "47744136-9332-4dc9-955a-7f25216a88fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:37:06.966599 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.966580    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47744136-9332-4dc9-955a-7f25216a88fd" (UID: "47744136-9332-4dc9-955a-7f25216a88fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:37:06.966754 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:06.966712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f" (OuterVolumeSpecName: "kube-api-access-7rt4f") pod "47744136-9332-4dc9-955a-7f25216a88fd" (UID: "47744136-9332-4dc9-955a-7f25216a88fd"). InnerVolumeSpecName "kube-api-access-7rt4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:37:07.066226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.066147    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47744136-9332-4dc9-955a-7f25216a88fd-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:37:07.066226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.066176    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rt4f\" (UniqueName: \"kubernetes.io/projected/47744136-9332-4dc9-955a-7f25216a88fd-kube-api-access-7rt4f\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:37:07.066226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.066186    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:37:07.066226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.066195    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47744136-9332-4dc9-955a-7f25216a88fd-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:37:07.066226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.066204    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47744136-9332-4dc9-955a-7f25216a88fd-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:37:07.167172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.167119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:07.169621 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.169592    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:07.289737 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.289692    2558 generic.go:356] "Generic (PLEG): container finished" podID="47744136-9332-4dc9-955a-7f25216a88fd" containerID="dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c" exitCode=0
Jul 28 13:37:07.289905 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.289762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerDied","Data":"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"}
Jul 28 13:37:07.289905 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.289776    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"
Jul 28 13:37:07.289905 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.289826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56" event={"ID":"47744136-9332-4dc9-955a-7f25216a88fd","Type":"ContainerDied","Data":"5d5ad3b45e0f9ba17b361b3bd1acfcc299c1b8a799eb4d07dbec25610855c245"}
Jul 28 13:37:07.289905 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.289861    2558 scope.go:117] "RemoveContainer" containerID="bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"
Jul 28 13:37:07.298783 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.298765    2558 scope.go:117] "RemoveContainer" containerID="dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"
Jul 28 13:37:07.305924 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.305903    2558 scope.go:117] "RemoveContainer" containerID="86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"
Jul 28 13:37:07.313141 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.313115    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:37:07.314158 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.314036    2558 scope.go:117] "RemoveContainer" containerID="bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"
Jul 28 13:37:07.314505 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:37:07.314366    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad\": container with ID starting with bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad not found: ID does not exist" containerID="bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"
Jul 28 13:37:07.314505 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.314431    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad"} err="failed to get container status \"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad\": rpc error: code = NotFound desc = could not find container \"bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad\": container with ID starting with bff0897938af19dea4e0760611f6fd405100f5ead7f5c412bb120f38dbb294ad not found: ID does not exist"
Jul 28 13:37:07.314666 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.314518    2558 scope.go:117] "RemoveContainer" containerID="dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"
Jul 28 13:37:07.314863 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:37:07.314843    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c\": container with ID starting with dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c not found: ID does not exist" containerID="dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"
Jul 28 13:37:07.314939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.314868    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c"} err="failed to get container status \"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c\": rpc error: code = NotFound desc = could not find container \"dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c\": container with ID starting with dfaf60204673de96bc36c9b267c6d9cba462e88b724dd434b55cb79ae225948c not found: ID does not exist"
Jul 28 13:37:07.314939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.314883    2558 scope.go:117] "RemoveContainer" containerID="86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"
Jul 28 13:37:07.315178 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:37:07.315144    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de\": container with ID starting with 86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de not found: ID does not exist" containerID="86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"
Jul 28 13:37:07.315254 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.315174    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de"} err="failed to get container status \"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de\": rpc error: code = NotFound desc = could not find container \"86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de\": container with ID starting with 86ff1612e62f15aa66f090a8068a5fa496f40836660e511f4afd72390d07d2de not found: ID does not exist"
Jul 28 13:37:07.316303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.316248    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-zdk56"]
Jul 28 13:37:07.373559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.373523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:07.501619 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:07.501524    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:37:07.504158 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:37:07.504131    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0deb03_2f41_4d27_8986_907042cef819.slice/crio-f5adb75c35e7837be4291e837742f12e05438f2cc1e103a54d6673304995e1fa WatchSource:0}: Error finding container f5adb75c35e7837be4291e837742f12e05438f2cc1e103a54d6673304995e1fa: Status 404 returned error can't find the container with id f5adb75c35e7837be4291e837742f12e05438f2cc1e103a54d6673304995e1fa
Jul 28 13:37:08.019765 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:08.019697    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47744136-9332-4dc9-955a-7f25216a88fd" path="/var/lib/kubelet/pods/47744136-9332-4dc9-955a-7f25216a88fd/volumes"
Jul 28 13:37:08.296078 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:08.295986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerStarted","Data":"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"}
Jul 28 13:37:08.296078 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:08.296042    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerStarted","Data":"f5adb75c35e7837be4291e837742f12e05438f2cc1e103a54d6673304995e1fa"}
Jul 28 13:37:11.310193 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:11.310161    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f0deb03-2f41-4d27-8986-907042cef819" containerID="d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013" exitCode=0
Jul 28 13:37:11.310654 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:11.310250    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerDied","Data":"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"}
Jul 28 13:37:12.316689 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:12.316652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerStarted","Data":"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"}
Jul 28 13:37:12.317091 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:12.316733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerStarted","Data":"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"}
Jul 28 13:37:12.317091 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:12.317044    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:12.317164 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:12.317095    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:12.337222 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:12.337180    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podStartSLOduration=2.630941977 podStartE2EDuration="6.337165177s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7062232s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:37:12.336514843 +0000 UTC m=+3738.822139436" watchObservedRunningTime="2026-07-28 13:37:12.337165177 +0000 UTC m=+3738.822789828"
Jul 28 13:37:18.327248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:18.327211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:37:48.329254 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:48.329157    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 13:37:58.328087 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:37:58.328039    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 13:38:08.328403 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:08.328359    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 13:38:18.328075 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:18.328032    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 13:38:22.089665 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:22.089625    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:38:27.453836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:27.453804    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:38:27.454384 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:27.454040    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" containerID="cri-o://d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5" gracePeriod=30
Jul 28 13:38:27.454384 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:27.454088    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kube-rbac-proxy" containerID="cri-o://834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd" gracePeriod=30
Jul 28 13:38:27.627256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:27.627221    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f0deb03-2f41-4d27-8986-907042cef819" containerID="834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd" exitCode=2
Jul 28 13:38:27.627256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:27.627257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerDied","Data":"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"}
Jul 28 13:38:28.322099 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:28.322048    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 28 13:38:32.013231 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.013189    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 13:38:32.306451 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.306427    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:38:32.466929 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.466893    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") pod \"9f0deb03-2f41-4d27-8986-907042cef819\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") "
Jul 28 13:38:32.467094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.466950    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert\") pod \"9f0deb03-2f41-4d27-8986-907042cef819\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") "
Jul 28 13:38:32.467094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467021    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8\") pod \"9f0deb03-2f41-4d27-8986-907042cef819\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") "
Jul 28 13:38:32.467094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467058    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config\") pod \"9f0deb03-2f41-4d27-8986-907042cef819\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") "
Jul 28 13:38:32.467232 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location\") pod \"9f0deb03-2f41-4d27-8986-907042cef819\" (UID: \"9f0deb03-2f41-4d27-8986-907042cef819\") "
Jul 28 13:38:32.467277 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467260    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f0deb03-2f41-4d27-8986-907042cef819" (UID: "9f0deb03-2f41-4d27-8986-907042cef819"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:32.467408 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467356    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:38:32.467484 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467466    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f0deb03-2f41-4d27-8986-907042cef819" (UID: "9f0deb03-2f41-4d27-8986-907042cef819"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:32.467548 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.467467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f0deb03-2f41-4d27-8986-907042cef819" (UID: "9f0deb03-2f41-4d27-8986-907042cef819"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:38:32.469263 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.469239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f0deb03-2f41-4d27-8986-907042cef819" (UID: "9f0deb03-2f41-4d27-8986-907042cef819"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:38:32.469428 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.469407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8" (OuterVolumeSpecName: "kube-api-access-dzhz8") pod "9f0deb03-2f41-4d27-8986-907042cef819" (UID: "9f0deb03-2f41-4d27-8986-907042cef819"). InnerVolumeSpecName "kube-api-access-dzhz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:38:32.568464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.568383    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/9f0deb03-2f41-4d27-8986-907042cef819-kube-api-access-dzhz8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:38:32.568464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.568412    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f0deb03-2f41-4d27-8986-907042cef819-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:38:32.568464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.568423    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f0deb03-2f41-4d27-8986-907042cef819-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:38:32.568464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.568431    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f0deb03-2f41-4d27-8986-907042cef819-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:38:32.618310 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618273    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:38:32.618580 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618569    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="storage-initializer"
Jul 28 13:38:32.618620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618582    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="storage-initializer"
Jul 28 13:38:32.618620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618592    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="storage-initializer"
Jul 28 13:38:32.618620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618597    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="storage-initializer"
Jul 28 13:38:32.618620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618611    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kube-rbac-proxy"
Jul 28 13:38:32.618620 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618617    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kube-rbac-proxy"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618623    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618629    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618641    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kube-rbac-proxy"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618646    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kube-rbac-proxy"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618652    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618657    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618702    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kube-rbac-proxy"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618709    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618730    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="47744136-9332-4dc9-955a-7f25216a88fd" containerName="kserve-container"
Jul 28 13:38:32.618795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.618737    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0deb03-2f41-4d27-8986-907042cef819" containerName="kube-rbac-proxy"
Jul 28 13:38:32.622036 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.622016    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.624527 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.624498    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 13:38:32.624640 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.624587    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:38:32.631483 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.631462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:38:32.649583 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.649554    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f0deb03-2f41-4d27-8986-907042cef819" containerID="d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5" exitCode=0
Jul 28 13:38:32.649707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.649602    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"
Jul 28 13:38:32.649707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.649634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerDied","Data":"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"}
Jul 28 13:38:32.649707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.649688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v" event={"ID":"9f0deb03-2f41-4d27-8986-907042cef819","Type":"ContainerDied","Data":"f5adb75c35e7837be4291e837742f12e05438f2cc1e103a54d6673304995e1fa"}
Jul 28 13:38:32.649908 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.649740    2558 scope.go:117] "RemoveContainer" containerID="834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"
Jul 28 13:38:32.658933 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.658914    2558 scope.go:117] "RemoveContainer" containerID="d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"
Jul 28 13:38:32.666163 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.666145    2558 scope.go:117] "RemoveContainer" containerID="d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"
Jul 28 13:38:32.671713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.671689    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:38:32.674073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.674053    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-44k7v"]
Jul 28 13:38:32.675029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675013    2558 scope.go:117] "RemoveContainer" containerID="834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"
Jul 28 13:38:32.675294 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:38:32.675274    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd\": container with ID starting with 834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd not found: ID does not exist" containerID="834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"
Jul 28 13:38:32.675339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675314    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd"} err="failed to get container status \"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd\": rpc error: code = NotFound desc = could not find container \"834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd\": container with ID starting with 834d0881da5dc9d25a7cebf877a1fd636e711f634f6214971c84237ceac4a0dd not found: ID does not exist"
Jul 28 13:38:32.675339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675331    2558 scope.go:117] "RemoveContainer" containerID="d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"
Jul 28 13:38:32.675590 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:38:32.675575    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5\": container with ID starting with d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5 not found: ID does not exist" containerID="d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"
Jul 28 13:38:32.675649 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675595    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5"} err="failed to get container status \"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5\": rpc error: code = NotFound desc = could not find container \"d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5\": container with ID starting with d35b77610a3ec897192be85404ddcb36d1dfb55142ad16a0fd405861cfe025c5 not found: ID does not exist"
Jul 28 13:38:32.675649 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675613    2558 scope.go:117] "RemoveContainer" containerID="d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"
Jul 28 13:38:32.675830 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:38:32.675812    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013\": container with ID starting with d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013 not found: ID does not exist" containerID="d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"
Jul 28 13:38:32.675872 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.675836    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013"} err="failed to get container status \"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013\": rpc error: code = NotFound desc = could not find container \"d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013\": container with ID starting with d19334e497a3b58f4083370674a8a8652b7651d4e2ebba6a8739b74becf20013 not found: ID does not exist"
Jul 28 13:38:32.770245 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.770217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ql8\" (UniqueName: \"kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.770436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.770263    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.770436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.770322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.770436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.770384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.770436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.770403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871111 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871076    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871111 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871560 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ql8\" (UniqueName: \"kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871560 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871560 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.871883 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.871854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.872163 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.872142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.872250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.872229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.874213 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.874192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.880527 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.880508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ql8\" (UniqueName: \"kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:32.933416 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:32.933384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:33.066991 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:33.066958    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:38:33.070133 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:38:33.070105    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc66081_734e_4d80_ba30_b7b4e188a587.slice/crio-3a20a1c8a5cc5e4ec3628cb0fbec5ce9ab9212c4ed69e74796ca1dcd6a225fc3 WatchSource:0}: Error finding container 3a20a1c8a5cc5e4ec3628cb0fbec5ce9ab9212c4ed69e74796ca1dcd6a225fc3: Status 404 returned error can't find the container with id 3a20a1c8a5cc5e4ec3628cb0fbec5ce9ab9212c4ed69e74796ca1dcd6a225fc3
Jul 28 13:38:33.656522 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:33.656483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerStarted","Data":"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"}
Jul 28 13:38:33.656708 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:33.656541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerStarted","Data":"3a20a1c8a5cc5e4ec3628cb0fbec5ce9ab9212c4ed69e74796ca1dcd6a225fc3"}
Jul 28 13:38:34.017314 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:34.017237    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0deb03-2f41-4d27-8986-907042cef819" path="/var/lib/kubelet/pods/9f0deb03-2f41-4d27-8986-907042cef819/volumes"
Jul 28 13:38:37.674672 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:37.674637    2558 generic.go:356] "Generic (PLEG): container finished" podID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerID="e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820" exitCode=0
Jul 28 13:38:37.675088 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:37.674710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerDied","Data":"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"}
Jul 28 13:38:38.689407 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:38.689347    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerStarted","Data":"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"}
Jul 28 13:38:38.689915 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:38.689447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerStarted","Data":"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"}
Jul 28 13:38:38.689915 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:38.689638    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:38.709311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:38.709259    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podStartSLOduration=3.135499063 podStartE2EDuration="6.709244553s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.57374549s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:38:38.708269343 +0000 UTC m=+3825.193893947" watchObservedRunningTime="2026-07-28 13:38:38.709244553 +0000 UTC m=+3825.194869144"
Jul 28 13:38:39.694070 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:39.694037    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:38:45.704397 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:38:45.704366    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:39:15.705402 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:15.705312    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 13:39:25.705364 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:25.705323    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 13:39:35.705850 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:35.705800    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 13:39:45.705763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:45.705695    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 13:39:55.178116 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:55.178001    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:39:55.193780 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:55.193752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:39:55.787422 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:39:55.787392    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:40:02.782223 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:02.782186    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:40:02.782786 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:02.782427    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" containerID="cri-o://922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02" gracePeriod=30
Jul 28 13:40:02.782786 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:02.782483    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kube-rbac-proxy" containerID="cri-o://1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0" gracePeriod=30
Jul 28 13:40:03.045956 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:03.045863    2558 generic.go:356] "Generic (PLEG): container finished" podID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerID="1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0" exitCode=2
Jul 28 13:40:03.045956 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:03.045937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerDied","Data":"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"}
Jul 28 13:40:05.698180 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:05.698132    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 13:40:05.704966 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:05.704935    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 13:40:07.534042 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.534011    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:40:07.638842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.638752    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config\") pod \"7bc66081-734e-4d80-ba30-b7b4e188a587\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") "
Jul 28 13:40:07.638842 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.638807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location\") pod \"7bc66081-734e-4d80-ba30-b7b4e188a587\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") "
Jul 28 13:40:07.639086 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.638844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls\") pod \"7bc66081-734e-4d80-ba30-b7b4e188a587\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") "
Jul 28 13:40:07.639086 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.638906    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46ql8\" (UniqueName: \"kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8\") pod \"7bc66081-734e-4d80-ba30-b7b4e188a587\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") "
Jul 28 13:40:07.639086 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.638961    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert\") pod \"7bc66081-734e-4d80-ba30-b7b4e188a587\" (UID: \"7bc66081-734e-4d80-ba30-b7b4e188a587\") "
Jul 28 13:40:07.639248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.639099    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bc66081-734e-4d80-ba30-b7b4e188a587" (UID: "7bc66081-734e-4d80-ba30-b7b4e188a587"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:40:07.639248 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.639161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bc66081-734e-4d80-ba30-b7b4e188a587" (UID: "7bc66081-734e-4d80-ba30-b7b4e188a587"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:40:07.639325 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.639171    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:40:07.639386 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.639362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bc66081-734e-4d80-ba30-b7b4e188a587" (UID: "7bc66081-734e-4d80-ba30-b7b4e188a587"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:40:07.641197 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.641166    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8" (OuterVolumeSpecName: "kube-api-access-46ql8") pod "7bc66081-734e-4d80-ba30-b7b4e188a587" (UID: "7bc66081-734e-4d80-ba30-b7b4e188a587"). InnerVolumeSpecName "kube-api-access-46ql8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:40:07.641197 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.641182    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bc66081-734e-4d80-ba30-b7b4e188a587" (UID: "7bc66081-734e-4d80-ba30-b7b4e188a587"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:40:07.740322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.740281    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc66081-734e-4d80-ba30-b7b4e188a587-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:40:07.740322 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.740317    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc66081-734e-4d80-ba30-b7b4e188a587-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:40:07.740533 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.740341    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc66081-734e-4d80-ba30-b7b4e188a587-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:40:07.740533 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.740352    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ql8\" (UniqueName: \"kubernetes.io/projected/7bc66081-734e-4d80-ba30-b7b4e188a587-kube-api-access-46ql8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:40:07.994416 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994324    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:40:07.994743 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994698    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container"
Jul 28 13:40:07.994743 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994734    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container"
Jul 28 13:40:07.994886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994752    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kube-rbac-proxy"
Jul 28 13:40:07.994886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994760    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kube-rbac-proxy"
Jul 28 13:40:07.994886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994781    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="storage-initializer"
Jul 28 13:40:07.994886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994790    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="storage-initializer"
Jul 28 13:40:07.994886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994876    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kserve-container"
Jul 28 13:40:07.995134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.994890    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerName="kube-rbac-proxy"
Jul 28 13:40:07.998380 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:07.998358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.002505 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.002477    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 13:40:08.002982 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.002965    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:40:08.020886 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.020851    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:40:08.069787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.069745    2558 generic.go:356] "Generic (PLEG): container finished" podID="7bc66081-734e-4d80-ba30-b7b4e188a587" containerID="922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02" exitCode=0
Jul 28 13:40:08.069939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.069798    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"
Jul 28 13:40:08.069939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.069812    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerDied","Data":"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"}
Jul 28 13:40:08.069939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.069884    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz" event={"ID":"7bc66081-734e-4d80-ba30-b7b4e188a587","Type":"ContainerDied","Data":"3a20a1c8a5cc5e4ec3628cb0fbec5ce9ab9212c4ed69e74796ca1dcd6a225fc3"}
Jul 28 13:40:08.069939 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.069921    2558 scope.go:117] "RemoveContainer" containerID="1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"
Jul 28 13:40:08.078217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.078196    2558 scope.go:117] "RemoveContainer" containerID="922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"
Jul 28 13:40:08.085259 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.085242    2558 scope.go:117] "RemoveContainer" containerID="e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"
Jul 28 13:40:08.091792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.091767    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:40:08.093863 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.093839    2558 scope.go:117] "RemoveContainer" containerID="1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"
Jul 28 13:40:08.094138 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:40:08.094119    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0\": container with ID starting with 1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0 not found: ID does not exist" containerID="1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"
Jul 28 13:40:08.094188 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.094150    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0"} err="failed to get container status \"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0\": rpc error: code = NotFound desc = could not find container \"1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0\": container with ID starting with 1dca1a35a454dc4f607a82d3e329fca332eaaf0704528e57c0aebe0dfb5a89a0 not found: ID does not exist"
Jul 28 13:40:08.094188 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.094167    2558 scope.go:117] "RemoveContainer" containerID="922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"
Jul 28 13:40:08.094410 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:40:08.094393    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02\": container with ID starting with 922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02 not found: ID does not exist" containerID="922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"
Jul 28 13:40:08.094485 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.094428    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02"} err="failed to get container status \"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02\": rpc error: code = NotFound desc = could not find container \"922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02\": container with ID starting with 922192cd33224231dfa937056362c9b66608d7ecacf229e96d2452b160fc3e02 not found: ID does not exist"
Jul 28 13:40:08.094485 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.094452    2558 scope.go:117] "RemoveContainer" containerID="e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"
Jul 28 13:40:08.094668 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:40:08.094653    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820\": container with ID starting with e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820 not found: ID does not exist" containerID="e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"
Jul 28 13:40:08.094711 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.094673    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820"} err="failed to get container status \"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820\": rpc error: code = NotFound desc = could not find container \"e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820\": container with ID starting with e5838b6f6cbeabb329614e1edcd36e199bf9895a0c193900a4968fd5e1787820 not found: ID does not exist"
Jul 28 13:40:08.101523 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.101497    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jrngz"]
Jul 28 13:40:08.144370 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.144337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.144499 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.144391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.144499 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.144471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95n7\" (UniqueName: \"kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.144593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.144501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.144593 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.144522    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246162 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246061    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246162 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246386 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95n7\" (UniqueName: \"kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246386 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246386 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:40:08.246237    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 13:40:08.246386 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246240    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246386 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:40:08.246322    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls podName:abe2a617-a020-4d82-87c0-ac90f5465020 nodeName:}" failed. No retries permitted until 2026-07-28 13:40:08.746299302 +0000 UTC m=+3915.231923894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 13:40:08.246634 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246896 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246878    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.246951 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.246933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.256211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.256190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95n7\" (UniqueName: \"kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.750422 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.750389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.753055 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.753034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:08.908239 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:08.908203    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:09.045920 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:09.045893    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:40:09.047592 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:40:09.047566    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe2a617_a020_4d82_87c0_ac90f5465020.slice/crio-681d969a3f6fa994d1ace884e5a3bf952ce602fcaf7b4d68ffa68f6c56b34816 WatchSource:0}: Error finding container 681d969a3f6fa994d1ace884e5a3bf952ce602fcaf7b4d68ffa68f6c56b34816: Status 404 returned error can't find the container with id 681d969a3f6fa994d1ace884e5a3bf952ce602fcaf7b4d68ffa68f6c56b34816
Jul 28 13:40:09.075063 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:09.075037    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerStarted","Data":"681d969a3f6fa994d1ace884e5a3bf952ce602fcaf7b4d68ffa68f6c56b34816"}
Jul 28 13:40:10.017492 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:10.017455    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc66081-734e-4d80-ba30-b7b4e188a587" path="/var/lib/kubelet/pods/7bc66081-734e-4d80-ba30-b7b4e188a587/volumes"
Jul 28 13:40:10.080666 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:10.080628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerStarted","Data":"84fbee0f12260287106957a0ee0c0e3846f3ebe3552a0ccf026097e2679bdab8"}
Jul 28 13:40:13.094321 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:13.094293    2558 generic.go:356] "Generic (PLEG): container finished" podID="abe2a617-a020-4d82-87c0-ac90f5465020" containerID="84fbee0f12260287106957a0ee0c0e3846f3ebe3552a0ccf026097e2679bdab8" exitCode=0
Jul 28 13:40:13.094779 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:13.094331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerDied","Data":"84fbee0f12260287106957a0ee0c0e3846f3ebe3552a0ccf026097e2679bdab8"}
Jul 28 13:40:14.100486 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:14.100451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerStarted","Data":"6ee00eeb75af87d016058d2592c60a34cef1abff21a4ca380787747fdc59d69b"}
Jul 28 13:40:14.100892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:14.100518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerStarted","Data":"2d9f9d78b396984460905682acb5b1fd80b7ccc9aa91bfe0c5810da10b69574d"}
Jul 28 13:40:14.100892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:14.100744    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:14.121557 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:14.121501    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podStartSLOduration=3.581327574 podStartE2EDuration="7.121486722s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.540159148s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:40:14.120142779 +0000 UTC m=+3920.605767371" watchObservedRunningTime="2026-07-28 13:40:14.121486722 +0000 UTC m=+3920.607111350"
Jul 28 13:40:15.104414 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:15.104385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:21.113941 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:21.113909    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:40:51.114934 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:40:51.114849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 13:41:01.115173 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:01.115133    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 13:41:11.114661 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:11.114621    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 13:41:21.115570 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:21.115520    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 13:41:31.187328 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:31.187291    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:41:38.101570 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:38.101527    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:41:38.102472 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:38.102432    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" containerID="cri-o://2d9f9d78b396984460905682acb5b1fd80b7ccc9aa91bfe0c5810da10b69574d" gracePeriod=30
Jul 28 13:41:38.102613 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:38.102476    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kube-rbac-proxy" containerID="cri-o://6ee00eeb75af87d016058d2592c60a34cef1abff21a4ca380787747fdc59d69b" gracePeriod=30
Jul 28 13:41:38.447839 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:38.447737    2558 generic.go:356] "Generic (PLEG): container finished" podID="abe2a617-a020-4d82-87c0-ac90f5465020" containerID="6ee00eeb75af87d016058d2592c60a34cef1abff21a4ca380787747fdc59d69b" exitCode=2
Jul 28 13:41:38.447839 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:38.447767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerDied","Data":"6ee00eeb75af87d016058d2592c60a34cef1abff21a4ca380787747fdc59d69b"}
Jul 28 13:41:41.109633 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:41.109582    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 28 13:41:41.114982 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:41.114957    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 13:41:43.471160 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.471127    2558 generic.go:356] "Generic (PLEG): container finished" podID="abe2a617-a020-4d82-87c0-ac90f5465020" containerID="2d9f9d78b396984460905682acb5b1fd80b7ccc9aa91bfe0c5810da10b69574d" exitCode=0
Jul 28 13:41:43.471605 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.471170    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerDied","Data":"2d9f9d78b396984460905682acb5b1fd80b7ccc9aa91bfe0c5810da10b69574d"}
Jul 28 13:41:43.550202 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.550179    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:41:43.701602 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701499    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location\") pod \"abe2a617-a020-4d82-87c0-ac90f5465020\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") "
Jul 28 13:41:43.701797 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") pod \"abe2a617-a020-4d82-87c0-ac90f5465020\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") "
Jul 28 13:41:43.701797 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701648    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert\") pod \"abe2a617-a020-4d82-87c0-ac90f5465020\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") "
Jul 28 13:41:43.701797 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701663    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config\") pod \"abe2a617-a020-4d82-87c0-ac90f5465020\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") "
Jul 28 13:41:43.701797 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w95n7\" (UniqueName: \"kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7\") pod \"abe2a617-a020-4d82-87c0-ac90f5465020\" (UID: \"abe2a617-a020-4d82-87c0-ac90f5465020\") "
Jul 28 13:41:43.701979 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.701884    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abe2a617-a020-4d82-87c0-ac90f5465020" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:41:43.702042 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.702026    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe2a617-a020-4d82-87c0-ac90f5465020-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:41:43.702105 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.702083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abe2a617-a020-4d82-87c0-ac90f5465020" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:41:43.702141 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.702091    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abe2a617-a020-4d82-87c0-ac90f5465020" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:41:43.703888 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.703866    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abe2a617-a020-4d82-87c0-ac90f5465020" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:41:43.703984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.703948    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7" (OuterVolumeSpecName: "kube-api-access-w95n7") pod "abe2a617-a020-4d82-87c0-ac90f5465020" (UID: "abe2a617-a020-4d82-87c0-ac90f5465020"). InnerVolumeSpecName "kube-api-access-w95n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:41:43.803067 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.803025    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe2a617-a020-4d82-87c0-ac90f5465020-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:41:43.803067 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.803058    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:41:43.803067 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.803068    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abe2a617-a020-4d82-87c0-ac90f5465020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:41:43.803067 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:43.803076    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95n7\" (UniqueName: \"kubernetes.io/projected/abe2a617-a020-4d82-87c0-ac90f5465020-kube-api-access-w95n7\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:41:44.477334 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.477304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t" event={"ID":"abe2a617-a020-4d82-87c0-ac90f5465020","Type":"ContainerDied","Data":"681d969a3f6fa994d1ace884e5a3bf952ce602fcaf7b4d68ffa68f6c56b34816"}
Jul 28 13:41:44.477334 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.477335    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"
Jul 28 13:41:44.477865 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.477369    2558 scope.go:117] "RemoveContainer" containerID="6ee00eeb75af87d016058d2592c60a34cef1abff21a4ca380787747fdc59d69b"
Jul 28 13:41:44.485757 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.485737    2558 scope.go:117] "RemoveContainer" containerID="2d9f9d78b396984460905682acb5b1fd80b7ccc9aa91bfe0c5810da10b69574d"
Jul 28 13:41:44.492752 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.492714    2558 scope.go:117] "RemoveContainer" containerID="84fbee0f12260287106957a0ee0c0e3846f3ebe3552a0ccf026097e2679bdab8"
Jul 28 13:41:44.498181 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.498157    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:41:44.501818 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:44.501794    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-2rb4t"]
Jul 28 13:41:46.017407 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:46.017372    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" path="/var/lib/kubelet/pods/abe2a617-a020-4d82-87c0-ac90f5465020/volumes"
Jul 28 13:41:50.453305 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453272    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453561    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453572    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453580    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kube-rbac-proxy"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453585    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kube-rbac-proxy"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453616    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="storage-initializer"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453625    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="storage-initializer"
Jul 28 13:41:50.453688 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453687    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kube-rbac-proxy"
Jul 28 13:41:50.453988 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.453699    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe2a617-a020-4d82-87c0-ac90f5465020" containerName="kserve-container"
Jul 28 13:41:50.458653 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.458634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.461751 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.461732    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:41:50.461751 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.461742    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:41:50.461918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.461766    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 13:41:50.462110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.462093    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 13:41:50.462182 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.462138    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:41:50.462182 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.462165    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:41:50.462993 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.462979    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:41:50.467583 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.467556    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:41:50.562293 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.562255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjn2\" (UniqueName: \"kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.562293 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.562293    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.562504 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.562317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.562504 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.562399    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.562574 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.562498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.663579 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.663544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.663695 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.663584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.663695 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.663650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.663695 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.663680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjn2\" (UniqueName: \"kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.663878 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.663706    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.664155 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.664132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.664352 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.664332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.664400 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.664344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.666424 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.666404    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.672549 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.672529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjn2\" (UniqueName: \"kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2\") pod \"isvc-sklearn-predictor-77b8bcdf88-sglgx\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.769560 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.769468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:50.903202 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:50.903177    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:41:50.904931 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:41:50.904902    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b54c649_af5f_44a5_a8ef_57a7fd33b89c.slice/crio-016fadfb22832df297a6c0e8bfc11f308e36c67dabeb7e1f89a4d0814520e4be WatchSource:0}: Error finding container 016fadfb22832df297a6c0e8bfc11f308e36c67dabeb7e1f89a4d0814520e4be: Status 404 returned error can't find the container with id 016fadfb22832df297a6c0e8bfc11f308e36c67dabeb7e1f89a4d0814520e4be
Jul 28 13:41:51.508152 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:51.508116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerStarted","Data":"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"}
Jul 28 13:41:51.508597 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:51.508174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerStarted","Data":"016fadfb22832df297a6c0e8bfc11f308e36c67dabeb7e1f89a4d0814520e4be"}
Jul 28 13:41:55.526216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:55.526183    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerID="6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd" exitCode=0
Jul 28 13:41:55.526602 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:55.526259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerDied","Data":"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"}
Jul 28 13:41:56.532385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.532355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerStarted","Data":"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"}
Jul 28 13:41:56.532824 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.532412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerStarted","Data":"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"}
Jul 28 13:41:56.532824 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.532692    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:56.532949 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.532863    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:41:56.534068 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.534044    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:41:56.551737 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:56.551674    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podStartSLOduration=2.67749256 podStartE2EDuration="6.55166136s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8741688s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:41:56.550568743 +0000 UTC m=+4023.036193336" watchObservedRunningTime="2026-07-28 13:41:56.55166136 +0000 UTC m=+4023.037285951"
Jul 28 13:41:57.536899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:41:57.536857    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:02.542398 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:02.542367    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:42:02.542878 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:02.542853    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:12.543073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:12.543032    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:22.543312 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:22.543274    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:32.542838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:32.542797    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:42.542973 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:42.542929    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:42:52.543086 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:42:52.543039    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:43:02.543287 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:02.543243    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:43:12.543709 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:12.543682    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:43:20.597806 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:20.597776    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:43:20.598254 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:20.598016    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" containerID="cri-o://cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5" gracePeriod=30
Jul 28 13:43:20.598254 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:20.598119    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kube-rbac-proxy" containerID="cri-o://7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4" gracePeriod=30
Jul 28 13:43:20.881763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:20.881645    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerID="7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4" exitCode=2
Jul 28 13:43:20.881763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:20.881739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerDied","Data":"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"}
Jul 28 13:43:22.537539 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:22.537487    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 28 13:43:22.542900 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:22.542867    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 13:43:25.351050 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.351025    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:43:25.423919 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.423881    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location\") pod \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") "
Jul 28 13:43:25.423919 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.423920    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config\") pod \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") "
Jul 28 13:43:25.424172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.423964    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls\") pod \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") "
Jul 28 13:43:25.424172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frjn2\" (UniqueName: \"kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2\") pod \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") "
Jul 28 13:43:25.424284 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424179    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert\") pod \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\" (UID: \"1b54c649-af5f-44a5-a8ef-57a7fd33b89c\") "
Jul 28 13:43:25.424284 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b54c649-af5f-44a5-a8ef-57a7fd33b89c" (UID: "1b54c649-af5f-44a5-a8ef-57a7fd33b89c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:43:25.424438 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b54c649-af5f-44a5-a8ef-57a7fd33b89c" (UID: "1b54c649-af5f-44a5-a8ef-57a7fd33b89c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:25.424500 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424425    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:43:25.424500 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.424460    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b54c649-af5f-44a5-a8ef-57a7fd33b89c" (UID: "1b54c649-af5f-44a5-a8ef-57a7fd33b89c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:25.426165 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.426144    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2" (OuterVolumeSpecName: "kube-api-access-frjn2") pod "1b54c649-af5f-44a5-a8ef-57a7fd33b89c" (UID: "1b54c649-af5f-44a5-a8ef-57a7fd33b89c"). InnerVolumeSpecName "kube-api-access-frjn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:43:25.426165 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.426160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b54c649-af5f-44a5-a8ef-57a7fd33b89c" (UID: "1b54c649-af5f-44a5-a8ef-57a7fd33b89c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:43:25.525874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.525790    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjn2\" (UniqueName: \"kubernetes.io/projected/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-api-access-frjn2\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:43:25.525874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.525817    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:43:25.525874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.525827    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:43:25.525874 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.525836    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b54c649-af5f-44a5-a8ef-57a7fd33b89c-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:43:25.905378 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.905340    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerID="cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5" exitCode=0
Jul 28 13:43:25.905565 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.905396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"
Jul 28 13:43:25.905565 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.905424    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerDied","Data":"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"}
Jul 28 13:43:25.905565 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.905490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx" event={"ID":"1b54c649-af5f-44a5-a8ef-57a7fd33b89c","Type":"ContainerDied","Data":"016fadfb22832df297a6c0e8bfc11f308e36c67dabeb7e1f89a4d0814520e4be"}
Jul 28 13:43:25.905565 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.905519    2558 scope.go:117] "RemoveContainer" containerID="7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"
Jul 28 13:43:25.914025 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.914011    2558 scope.go:117] "RemoveContainer" containerID="cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"
Jul 28 13:43:25.921275 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.921251    2558 scope.go:117] "RemoveContainer" containerID="6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"
Jul 28 13:43:25.928274 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.928258    2558 scope.go:117] "RemoveContainer" containerID="7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"
Jul 28 13:43:25.928550 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:43:25.928526    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4\": container with ID starting with 7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4 not found: ID does not exist" containerID="7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"
Jul 28 13:43:25.928638 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.928558    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4"} err="failed to get container status \"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4\": rpc error: code = NotFound desc = could not find container \"7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4\": container with ID starting with 7b124b02a246b7c2fdf82ed7ad2a7039074fe804c005e0d00f020f28644ed7a4 not found: ID does not exist"
Jul 28 13:43:25.928638 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.928574    2558 scope.go:117] "RemoveContainer" containerID="cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"
Jul 28 13:43:25.928926 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:43:25.928906    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5\": container with ID starting with cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5 not found: ID does not exist" containerID="cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"
Jul 28 13:43:25.929008 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.928937    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5"} err="failed to get container status \"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5\": rpc error: code = NotFound desc = could not find container \"cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5\": container with ID starting with cc99e898f13b76c4c32d5211040d2f99cea459dbeb1f1115516e569b5d34d2f5 not found: ID does not exist"
Jul 28 13:43:25.929008 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.928957    2558 scope.go:117] "RemoveContainer" containerID="6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"
Jul 28 13:43:25.929218 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:43:25.929201    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd\": container with ID starting with 6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd not found: ID does not exist" containerID="6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"
Jul 28 13:43:25.929255 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.929223    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd"} err="failed to get container status \"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd\": rpc error: code = NotFound desc = could not find container \"6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd\": container with ID starting with 6182b0edeff8f3fbdbdf970e63e8f0cb426187656d707981b538e80ef325aedd not found: ID does not exist"
Jul 28 13:43:25.938702 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.938668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:43:25.940180 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:25.940160    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-77b8bcdf88-sglgx"]
Jul 28 13:43:26.021549 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:43:26.021519    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" path="/var/lib/kubelet/pods/1b54c649-af5f-44a5-a8ef-57a7fd33b89c/volumes"
Jul 28 13:44:31.038648 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038609    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038923    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kube-rbac-proxy"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038933    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kube-rbac-proxy"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038947    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="storage-initializer"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038952    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="storage-initializer"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038974    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.038983    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.039034    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kube-rbac-proxy"
Jul 28 13:44:31.039256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.039040    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b54c649-af5f-44a5-a8ef-57a7fd33b89c" containerName="kserve-container"
Jul 28 13:44:31.042017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.042000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.045017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.044991    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:44:31.045017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.045009    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:44:31.045017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.045015    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:44:31.045288 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.044993    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:44:31.046041 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.046018    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 13:44:31.046186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.046040    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:44:31.046186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.046149    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:44:31.055891 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.055868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:44:31.149064 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.149021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.149235 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.149079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.149235 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.149106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.149235 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.149140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwkg\" (UniqueName: \"kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.149235 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.149189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250336 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250336 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250594 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250594 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwkg\" (UniqueName: \"kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250594 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.250594 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:44:31.250571    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 28 13:44:31.250834 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:44:31.250649    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls podName:067ab261-b6cc-46c5-9c94-1af536913cdc nodeName:}" failed. No retries permitted until 2026-07-28 13:44:31.750628651 +0000 UTC m=+4178.236253225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls") pod "isvc-sklearn-runtime-predictor-646574df76-dnb5n" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 28 13:44:31.250920 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.250897    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.251024 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.251007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.251115 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.251099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.261411 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.261381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwkg\" (UniqueName: \"kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.754067 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.754026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.756581 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.756560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-646574df76-dnb5n\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:31.952338 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:31.952293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:32.086397 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:32.086369    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:44:32.089156 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:44:32.089130    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ab261_b6cc_46c5_9c94_1af536913cdc.slice/crio-65236dcb6df821acd643ded1dc4ffb14f6f76d8c7bd1dd4b951323f3330a7cd5 WatchSource:0}: Error finding container 65236dcb6df821acd643ded1dc4ffb14f6f76d8c7bd1dd4b951323f3330a7cd5: Status 404 returned error can't find the container with id 65236dcb6df821acd643ded1dc4ffb14f6f76d8c7bd1dd4b951323f3330a7cd5
Jul 28 13:44:32.171568 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:32.171539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerStarted","Data":"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"}
Jul 28 13:44:32.171742 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:32.171593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerStarted","Data":"65236dcb6df821acd643ded1dc4ffb14f6f76d8c7bd1dd4b951323f3330a7cd5"}
Jul 28 13:44:38.196685 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:38.196655    2558 generic.go:356] "Generic (PLEG): container finished" podID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerID="9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42" exitCode=0
Jul 28 13:44:38.197090 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:38.196749    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerDied","Data":"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"}
Jul 28 13:44:39.203522 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:39.203483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerStarted","Data":"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"}
Jul 28 13:44:39.203933 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:39.203568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerStarted","Data":"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"}
Jul 28 13:44:39.203933 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:39.203893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:39.222711 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:39.222661    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podStartSLOduration=2.6168721980000003 podStartE2EDuration="8.222648596s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.605776398s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:39.221793847 +0000 UTC m=+4185.707418441" watchObservedRunningTime="2026-07-28 13:44:39.222648596 +0000 UTC m=+4185.708273188"
Jul 28 13:44:40.208122 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:40.208091    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:40.209525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:40.209501    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 13:44:41.212596 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:41.212555    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 13:44:46.217203 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:46.217174    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:44:46.217698 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:46.217670    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 13:44:55.202917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:55.202794    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:44:55.223550 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:55.223529    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:44:56.218586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:44:56.218558    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:45:02.830681 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:02.830644    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-646574df76-dnb5n_067ab261-b6cc-46c5-9c94-1af536913cdc/kserve-container/0.log"
Jul 28 13:45:02.972192 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:02.972159    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:45:02.972407 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:02.972377    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container" containerID="cri-o://3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21" gracePeriod=30
Jul 28 13:45:02.972523 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:02.972435    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kube-rbac-proxy" containerID="cri-o://356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211" gracePeriod=30
Jul 28 13:45:03.309445 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:03.309404    2558 generic.go:356] "Generic (PLEG): container finished" podID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerID="356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211" exitCode=2
Jul 28 13:45:03.309637 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:03.309460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerDied","Data":"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"}
Jul 28 13:45:04.015178 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.015149    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:45:04.135230 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135135    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hwkg\" (UniqueName: \"kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg\") pod \"067ab261-b6cc-46c5-9c94-1af536913cdc\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") "
Jul 28 13:45:04.135404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert\") pod \"067ab261-b6cc-46c5-9c94-1af536913cdc\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") "
Jul 28 13:45:04.135404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135285    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location\") pod \"067ab261-b6cc-46c5-9c94-1af536913cdc\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") "
Jul 28 13:45:04.135404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135327    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config\") pod \"067ab261-b6cc-46c5-9c94-1af536913cdc\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") "
Jul 28 13:45:04.135404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135360    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") pod \"067ab261-b6cc-46c5-9c94-1af536913cdc\" (UID: \"067ab261-b6cc-46c5-9c94-1af536913cdc\") "
Jul 28 13:45:04.135670 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135639    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "067ab261-b6cc-46c5-9c94-1af536913cdc" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:45:04.135750 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.135670    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "067ab261-b6cc-46c5-9c94-1af536913cdc" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:45:04.137494 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.137468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg" (OuterVolumeSpecName: "kube-api-access-9hwkg") pod "067ab261-b6cc-46c5-9c94-1af536913cdc" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc"). InnerVolumeSpecName "kube-api-access-9hwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:45:04.137494 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.137490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "067ab261-b6cc-46c5-9c94-1af536913cdc" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:45:04.160156 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.160115    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "067ab261-b6cc-46c5-9c94-1af536913cdc" (UID: "067ab261-b6cc-46c5-9c94-1af536913cdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:45:04.236877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.236843    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:45:04.236877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.236875    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/067ab261-b6cc-46c5-9c94-1af536913cdc-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:45:04.237084 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.236890    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:45:04.237084 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.236903    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/067ab261-b6cc-46c5-9c94-1af536913cdc-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:45:04.237084 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.236917    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwkg\" (UniqueName: \"kubernetes.io/projected/067ab261-b6cc-46c5-9c94-1af536913cdc-kube-api-access-9hwkg\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:45:04.316282 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.316246    2558 generic.go:356] "Generic (PLEG): container finished" podID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerID="3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21" exitCode=0
Jul 28 13:45:04.316442 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.316300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerDied","Data":"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"}
Jul 28 13:45:04.316442 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.316352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n" event={"ID":"067ab261-b6cc-46c5-9c94-1af536913cdc","Type":"ContainerDied","Data":"65236dcb6df821acd643ded1dc4ffb14f6f76d8c7bd1dd4b951323f3330a7cd5"}
Jul 28 13:45:04.316442 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.316380    2558 scope.go:117] "RemoveContainer" containerID="356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"
Jul 28 13:45:04.316442 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.316303    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"
Jul 28 13:45:04.324775 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.324756    2558 scope.go:117] "RemoveContainer" containerID="3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"
Jul 28 13:45:04.331750 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.331713    2558 scope.go:117] "RemoveContainer" containerID="9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"
Jul 28 13:45:04.338915 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.338892    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:45:04.339424 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.339406    2558 scope.go:117] "RemoveContainer" containerID="356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"
Jul 28 13:45:04.339666 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:45:04.339647    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211\": container with ID starting with 356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211 not found: ID does not exist" containerID="356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"
Jul 28 13:45:04.339734 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.339677    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211"} err="failed to get container status \"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211\": rpc error: code = NotFound desc = could not find container \"356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211\": container with ID starting with 356cee1e4736d7e42d0473f037908b879e1c25053e668834c21b217d7a6f4211 not found: ID does not exist"
Jul 28 13:45:04.339734 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.339693    2558 scope.go:117] "RemoveContainer" containerID="3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"
Jul 28 13:45:04.339946 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:45:04.339927    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21\": container with ID starting with 3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21 not found: ID does not exist" containerID="3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"
Jul 28 13:45:04.340010 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.339956    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21"} err="failed to get container status \"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21\": rpc error: code = NotFound desc = could not find container \"3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21\": container with ID starting with 3c03c2a2e3efb2fd4f49da506e38e37d3843385ffc0ab505755356a97fd1cd21 not found: ID does not exist"
Jul 28 13:45:04.340010 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.339981    2558 scope.go:117] "RemoveContainer" containerID="9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"
Jul 28 13:45:04.340226 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:45:04.340208    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42\": container with ID starting with 9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42 not found: ID does not exist" containerID="9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"
Jul 28 13:45:04.340284 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.340251    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42"} err="failed to get container status \"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42\": rpc error: code = NotFound desc = could not find container \"9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42\": container with ID starting with 9c3e8af9b49e757b95057cb4ee7c367ed399b0859d3a7eec38949c36e002fd42 not found: ID does not exist"
Jul 28 13:45:04.344243 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:04.344223    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-646574df76-dnb5n"]
Jul 28 13:45:06.018436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:45:06.018382    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" path="/var/lib/kubelet/pods/067ab261-b6cc-46c5-9c94-1af536913cdc/volumes"
Jul 28 13:46:13.404029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.403990    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404350    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kube-rbac-proxy"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404365    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kube-rbac-proxy"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404378    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="storage-initializer"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404383    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="storage-initializer"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404393    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404399    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404466    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kserve-container"
Jul 28 13:46:13.404592 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.404473    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ab261-b6cc-46c5-9c94-1af536913cdc" containerName="kube-rbac-proxy"
Jul 28 13:46:13.407560 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.407546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.410346 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.410321    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:46:13.410493 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.410361    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:46:13.411594 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.411575    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:46:13.411748 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.411577    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:46:13.411748 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.411586    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:46:13.411748 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.411588    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 13:46:13.411974 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.411590    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:46:13.417110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.417087    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:46:13.550500 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.550457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.550704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.550509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.550704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.550577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ls8\" (UniqueName: \"kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.550704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.550609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.550704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.550647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.651647 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.651604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ls8\" (UniqueName: \"kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.651647 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.651647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.651930 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.651671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.651930 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.651700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.651930 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.651751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.652129 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.652110    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.652390 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.652372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.652457 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.652442    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.654299 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.654252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.660548 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.660516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ls8\" (UniqueName: \"kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8\") pod \"isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.718360 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.718326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:13.850112 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:13.850085    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:46:13.852025 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:46:13.851995    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d072783_68b3_4dc1_b6e1_6556389918d7.slice/crio-f558bace27fc3e359e5d42779e9e3c156f51638b152dcf03e1500f24a2fbcf68 WatchSource:0}: Error finding container f558bace27fc3e359e5d42779e9e3c156f51638b152dcf03e1500f24a2fbcf68: Status 404 returned error can't find the container with id f558bace27fc3e359e5d42779e9e3c156f51638b152dcf03e1500f24a2fbcf68
Jul 28 13:46:14.607707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:14.607669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerStarted","Data":"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"}
Jul 28 13:46:14.608124 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:14.607753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerStarted","Data":"f558bace27fc3e359e5d42779e9e3c156f51638b152dcf03e1500f24a2fbcf68"}
Jul 28 13:46:17.621338 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:17.621306    2558 generic.go:356] "Generic (PLEG): container finished" podID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerID="73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4" exitCode=0
Jul 28 13:46:17.621678 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:17.621376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerDied","Data":"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"}
Jul 28 13:46:18.627987 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.627948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerStarted","Data":"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"}
Jul 28 13:46:18.628411 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.628012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerStarted","Data":"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"}
Jul 28 13:46:18.628411 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.628279    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:18.628537 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.628417    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:18.629814 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.629786    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:46:18.649038 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:18.648988    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podStartSLOduration=2.134765897 podStartE2EDuration="5.648975239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.514209342s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:46:18.647373566 +0000 UTC m=+4285.132998184" watchObservedRunningTime="2026-07-28 13:46:18.648975239 +0000 UTC m=+4285.134599830"
Jul 28 13:46:19.631968 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:19.631918    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:46:24.636525 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:24.636496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:46:24.637152 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:24.637125    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:46:34.637075 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:34.637031    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:46:44.637381 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:44.637340    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:46:54.637219 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:46:54.637123    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:47:04.637658 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:04.637612    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:47:14.637699 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:14.637657    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:47:24.637685 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:24.637641    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:47:34.638524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:34.638491    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:47:43.654046 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:43.653994    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:47:43.654492 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:43.654248    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" containerID="cri-o://c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04" gracePeriod=30
Jul 28 13:47:43.654492 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:43.654287    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kube-rbac-proxy" containerID="cri-o://37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756" gracePeriod=30
Jul 28 13:47:43.968917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:43.968826    2558 generic.go:356] "Generic (PLEG): container finished" podID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerID="37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756" exitCode=2
Jul 28 13:47:43.968917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:43.968904    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerDied","Data":"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"}
Jul 28 13:47:44.632770 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:44.632707    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 28 13:47:44.637079 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:44.637055    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 13:47:48.502542 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.502517    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:47:48.525239 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls\") pod \"6d072783-68b3-4dc1-b6e1-6556389918d7\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") "
Jul 28 13:47:48.525402 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525251    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location\") pod \"6d072783-68b3-4dc1-b6e1-6556389918d7\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") "
Jul 28 13:47:48.525402 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525270    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5ls8\" (UniqueName: \"kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8\") pod \"6d072783-68b3-4dc1-b6e1-6556389918d7\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") "
Jul 28 13:47:48.525402 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config\") pod \"6d072783-68b3-4dc1-b6e1-6556389918d7\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") "
Jul 28 13:47:48.525402 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525339    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert\") pod \"6d072783-68b3-4dc1-b6e1-6556389918d7\" (UID: \"6d072783-68b3-4dc1-b6e1-6556389918d7\") "
Jul 28 13:47:48.525616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525579    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d072783-68b3-4dc1-b6e1-6556389918d7" (UID: "6d072783-68b3-4dc1-b6e1-6556389918d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:47:48.525707 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525685    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d072783-68b3-4dc1-b6e1-6556389918d7" (UID: "6d072783-68b3-4dc1-b6e1-6556389918d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:47:48.525782 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.525758    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d072783-68b3-4dc1-b6e1-6556389918d7" (UID: "6d072783-68b3-4dc1-b6e1-6556389918d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:47:48.527519 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.527497    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d072783-68b3-4dc1-b6e1-6556389918d7" (UID: "6d072783-68b3-4dc1-b6e1-6556389918d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:47:48.527644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.527599    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8" (OuterVolumeSpecName: "kube-api-access-k5ls8") pod "6d072783-68b3-4dc1-b6e1-6556389918d7" (UID: "6d072783-68b3-4dc1-b6e1-6556389918d7"). InnerVolumeSpecName "kube-api-access-k5ls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:47:48.625792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.625759    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:47:48.625792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.625785    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d072783-68b3-4dc1-b6e1-6556389918d7-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:47:48.625792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.625794    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d072783-68b3-4dc1-b6e1-6556389918d7-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:47:48.625792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.625802    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d072783-68b3-4dc1-b6e1-6556389918d7-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:47:48.626056 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.625811    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ls8\" (UniqueName: \"kubernetes.io/projected/6d072783-68b3-4dc1-b6e1-6556389918d7-kube-api-access-k5ls8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:47:48.896894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.896814    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:47:48.897133 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897121    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container"
Jul 28 13:47:48.897172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897135    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container"
Jul 28 13:47:48.897172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897145    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="storage-initializer"
Jul 28 13:47:48.897172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897151    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="storage-initializer"
Jul 28 13:47:48.897172 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897168    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kube-rbac-proxy"
Jul 28 13:47:48.897295 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897173    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kube-rbac-proxy"
Jul 28 13:47:48.897295 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897222    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kserve-container"
Jul 28 13:47:48.897295 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.897233    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerName="kube-rbac-proxy"
Jul 28 13:47:48.900436 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.900414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.904483 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.904458    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 13:47:48.904602 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.904506    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 13:47:48.920203 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.920176    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:47:48.928404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.928373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.928547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.928422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.928547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.928490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9npt\" (UniqueName: \"kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.928547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.928536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.928677 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.928565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:48.988660 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.988627    2558 generic.go:356] "Generic (PLEG): container finished" podID="6d072783-68b3-4dc1-b6e1-6556389918d7" containerID="c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04" exitCode=0
Jul 28 13:47:48.988860 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.988670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerDied","Data":"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"}
Jul 28 13:47:48.988860 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.988695    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"
Jul 28 13:47:48.988860 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.988739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm" event={"ID":"6d072783-68b3-4dc1-b6e1-6556389918d7","Type":"ContainerDied","Data":"f558bace27fc3e359e5d42779e9e3c156f51638b152dcf03e1500f24a2fbcf68"}
Jul 28 13:47:48.988860 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.988769    2558 scope.go:117] "RemoveContainer" containerID="37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"
Jul 28 13:47:48.997170 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:48.997149    2558 scope.go:117] "RemoveContainer" containerID="c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"
Jul 28 13:47:49.004332 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.004289    2558 scope.go:117] "RemoveContainer" containerID="73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"
Jul 28 13:47:49.011966 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.011950    2558 scope.go:117] "RemoveContainer" containerID="37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"
Jul 28 13:47:49.012226 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:47:49.012200    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756\": container with ID starting with 37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756 not found: ID does not exist" containerID="37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"
Jul 28 13:47:49.012335 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012231    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756"} err="failed to get container status \"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756\": rpc error: code = NotFound desc = could not find container \"37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756\": container with ID starting with 37f6cb21161ab0567f7bda0acb7091a1b57d801101c3cf02e891e6c50080d756 not found: ID does not exist"
Jul 28 13:47:49.012335 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012249    2558 scope.go:117] "RemoveContainer" containerID="c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"
Jul 28 13:47:49.012508 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:47:49.012492    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04\": container with ID starting with c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04 not found: ID does not exist" containerID="c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"
Jul 28 13:47:49.012552 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012514    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04"} err="failed to get container status \"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04\": rpc error: code = NotFound desc = could not find container \"c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04\": container with ID starting with c075c262a28110682c1d04abe7dc809d0b21874016c487850dfb347c7586db04 not found: ID does not exist"
Jul 28 13:47:49.012552 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012529    2558 scope.go:117] "RemoveContainer" containerID="73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"
Jul 28 13:47:49.012686 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012664    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:47:49.012912 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:47:49.012884    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4\": container with ID starting with 73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4 not found: ID does not exist" containerID="73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"
Jul 28 13:47:49.012983 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.012939    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4"} err="failed to get container status \"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4\": rpc error: code = NotFound desc = could not find container \"73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4\": container with ID starting with 73645263b392030b762f43c0c689c5b37cc464d9732d3a5f752e82798626fac4 not found: ID does not exist"
Jul 28 13:47:49.015247 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.015227    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-699bc6b4d4-5xbmm"]
Jul 28 13:47:49.029002 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.028982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9npt\" (UniqueName: \"kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029052    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029110 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029269 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029641 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029770 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.029929 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.029911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.031540 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.031525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.038101 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.038076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9npt\" (UniqueName: \"kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt\") pod \"isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.210146 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.210055    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:49.366483 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.366456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:47:49.369359 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:47:49.369333    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac25ea_d89b_4671_b83b_a7be7ef491ee.slice/crio-73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323 WatchSource:0}: Error finding container 73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323: Status 404 returned error can't find the container with id 73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323
Jul 28 13:47:49.994393 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.994357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerStarted","Data":"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"}
Jul 28 13:47:49.994890 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:49.994433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerStarted","Data":"73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323"}
Jul 28 13:47:50.017113 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:50.017080    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d072783-68b3-4dc1-b6e1-6556389918d7" path="/var/lib/kubelet/pods/6d072783-68b3-4dc1-b6e1-6556389918d7/volumes"
Jul 28 13:47:54.013174 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:54.013139    2558 generic.go:356] "Generic (PLEG): container finished" podID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerID="ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43" exitCode=0
Jul 28 13:47:54.016894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:54.016863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerDied","Data":"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"}
Jul 28 13:47:55.018473 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.018438    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerStarted","Data":"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"}
Jul 28 13:47:55.018916 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.018498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerStarted","Data":"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"}
Jul 28 13:47:55.018916 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.018813    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:55.018916 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.018860    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:47:55.020329 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.020294    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:47:55.043978 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:55.043922    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podStartSLOduration=3.256968208 podStartE2EDuration="7.043908642s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786940434s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:47:55.042451292 +0000 UTC m=+4381.528075877" watchObservedRunningTime="2026-07-28 13:47:55.043908642 +0000 UTC m=+4381.529533346"
Jul 28 13:47:56.022095 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:47:56.022056    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:01.026335 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:01.026305    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:48:01.026861 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:01.026805    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:11.027225 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:11.027186    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:21.027100 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:21.027042    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:31.027657 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:31.027616    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:41.027682 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:41.027643    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:48:51.027304 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:48:51.027261    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:49:01.026742 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:01.026677    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:49:11.027389 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:11.027358    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:49:18.965877 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:18.965830    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:49:18.966458 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:18.966072    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" containerID="cri-o://dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33" gracePeriod=30
Jul 28 13:49:18.966458 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:18.966177    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kube-rbac-proxy" containerID="cri-o://e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3" gracePeriod=30
Jul 28 13:49:19.368282 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:19.368247    2558 generic.go:356] "Generic (PLEG): container finished" podID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerID="e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3" exitCode=2
Jul 28 13:49:19.368455 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:19.368319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerDied","Data":"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"}
Jul 28 13:49:21.023049 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:21.023006    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 28 13:49:21.027315 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:21.027282    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 13:49:23.813677 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.813654    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:49:23.887261 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887232    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9npt\" (UniqueName: \"kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt\") pod \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") "
Jul 28 13:49:23.887464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887279    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config\") pod \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") "
Jul 28 13:49:23.887464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887312    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls\") pod \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") "
Jul 28 13:49:23.887464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location\") pod \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") "
Jul 28 13:49:23.887464 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887419    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert\") pod \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\" (UID: \"ecac25ea-d89b-4671-b83b-a7be7ef491ee\") "
Jul 28 13:49:23.887704 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecac25ea-d89b-4671-b83b-a7be7ef491ee" (UID: "ecac25ea-d89b-4671-b83b-a7be7ef491ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:49:23.887838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecac25ea-d89b-4671-b83b-a7be7ef491ee" (UID: "ecac25ea-d89b-4671-b83b-a7be7ef491ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:49:23.887899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.887881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecac25ea-d89b-4671-b83b-a7be7ef491ee" (UID: "ecac25ea-d89b-4671-b83b-a7be7ef491ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:49:23.889586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.889558    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecac25ea-d89b-4671-b83b-a7be7ef491ee" (UID: "ecac25ea-d89b-4671-b83b-a7be7ef491ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:49:23.889690 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.889611    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt" (OuterVolumeSpecName: "kube-api-access-h9npt") pod "ecac25ea-d89b-4671-b83b-a7be7ef491ee" (UID: "ecac25ea-d89b-4671-b83b-a7be7ef491ee"). InnerVolumeSpecName "kube-api-access-h9npt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:49:23.988760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.988702    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9npt\" (UniqueName: \"kubernetes.io/projected/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-api-access-h9npt\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:49:23.988760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.988755    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:49:23.988760 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.988765    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecac25ea-d89b-4671-b83b-a7be7ef491ee-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:49:23.989017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.988774    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecac25ea-d89b-4671-b83b-a7be7ef491ee-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:49:23.989017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:23.988784    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecac25ea-d89b-4671-b83b-a7be7ef491ee-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:49:24.061015 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:49:24.060980    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac25ea_d89b_4671_b83b_a7be7ef491ee.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecac25ea_d89b_4671_b83b_a7be7ef491ee.slice/crio-73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323\": RecentStats: unable to find data in memory cache]"
Jul 28 13:49:24.131685 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.131647    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:49:24.132019 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.131996    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kube-rbac-proxy"
Jul 28 13:49:24.132019 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132015    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kube-rbac-proxy"
Jul 28 13:49:24.132019 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132027    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container"
Jul 28 13:49:24.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132032    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container"
Jul 28 13:49:24.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132040    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="storage-initializer"
Jul 28 13:49:24.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132048    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="storage-initializer"
Jul 28 13:49:24.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132102    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kube-rbac-proxy"
Jul 28 13:49:24.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.132110    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerName="kserve-container"
Jul 28 13:49:24.135374 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.135358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.138226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.138155    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 13:49:24.138226 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.138180    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 13:49:24.146466 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.146439    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:49:24.190430 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.190397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llbw\" (UniqueName: \"kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.190611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.190487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.190611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.190510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.190611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.190538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.190611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.190560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291332 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291332 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291571 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291571 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291571 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llbw\" (UniqueName: \"kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.291792 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.291769    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.292053 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.292032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.292118 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.292085    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.293986 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.293970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.299955 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.299935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llbw\" (UniqueName: \"kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw\") pod \"isvc-tensorflow-predictor-547dfc5669-b9jqj\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.391419 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.391338    2558 generic.go:356] "Generic (PLEG): container finished" podID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" containerID="dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33" exitCode=0
Jul 28 13:49:24.391419 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.391383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerDied","Data":"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"}
Jul 28 13:49:24.391584 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.391420    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"
Jul 28 13:49:24.391584 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.391443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd" event={"ID":"ecac25ea-d89b-4671-b83b-a7be7ef491ee","Type":"ContainerDied","Data":"73f6318e567fcefa070fb8a743bf379c1cd431796e7c5f301e04b3cc9da00323"}
Jul 28 13:49:24.391584 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.391481    2558 scope.go:117] "RemoveContainer" containerID="e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"
Jul 28 13:49:24.400078 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.400057    2558 scope.go:117] "RemoveContainer" containerID="dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"
Jul 28 13:49:24.407672 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.407649    2558 scope.go:117] "RemoveContainer" containerID="ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"
Jul 28 13:49:24.409795 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.409774    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:49:24.415603 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.415577    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-845f7c8785-7sbwd"]
Jul 28 13:49:24.416009 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.415996    2558 scope.go:117] "RemoveContainer" containerID="e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"
Jul 28 13:49:24.416279 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:49:24.416259    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3\": container with ID starting with e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3 not found: ID does not exist" containerID="e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"
Jul 28 13:49:24.416318 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.416292    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3"} err="failed to get container status \"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3\": rpc error: code = NotFound desc = could not find container \"e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3\": container with ID starting with e13af52af820a5cb991b035254c572faf5d2778eb62c2108c3c685212a291db3 not found: ID does not exist"
Jul 28 13:49:24.416359 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.416318    2558 scope.go:117] "RemoveContainer" containerID="dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"
Jul 28 13:49:24.416550 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:49:24.416533    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33\": container with ID starting with dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33 not found: ID does not exist" containerID="dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"
Jul 28 13:49:24.416590 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.416557    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33"} err="failed to get container status \"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33\": rpc error: code = NotFound desc = could not find container \"dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33\": container with ID starting with dec007a02b6d24be6d599f0cef329d0a102bdb50dab428dec0bf436a378b6c33 not found: ID does not exist"
Jul 28 13:49:24.416590 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.416575    2558 scope.go:117] "RemoveContainer" containerID="ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"
Jul 28 13:49:24.416915 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:49:24.416900    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43\": container with ID starting with ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43 not found: ID does not exist" containerID="ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"
Jul 28 13:49:24.416979 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.416925    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43"} err="failed to get container status \"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43\": rpc error: code = NotFound desc = could not find container \"ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43\": container with ID starting with ab86cda7bb31838e96e79ce31ab445fcafbf72f631c37e9e56befd5c2fbf1c43 not found: ID does not exist"
Jul 28 13:49:24.445191 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.445156    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:24.586761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:24.586708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:49:24.590303 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:49:24.590273    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e0fe96_db34_43c0_90d8_8192a1088a97.slice/crio-365cafdfb05f6f10c89d1b5fc3e72ab8449359028ff9171c0ef5001669e57c01 WatchSource:0}: Error finding container 365cafdfb05f6f10c89d1b5fc3e72ab8449359028ff9171c0ef5001669e57c01: Status 404 returned error can't find the container with id 365cafdfb05f6f10c89d1b5fc3e72ab8449359028ff9171c0ef5001669e57c01
Jul 28 13:49:25.396780 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:25.396741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerStarted","Data":"d69a2abea6923f9bdb4b2a9c422cc3a2f8cba3f5de7da63ee7e1afeae8b1e210"}
Jul 28 13:49:25.397229 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:25.396810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerStarted","Data":"365cafdfb05f6f10c89d1b5fc3e72ab8449359028ff9171c0ef5001669e57c01"}
Jul 28 13:49:26.017061 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:26.017029    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecac25ea-d89b-4671-b83b-a7be7ef491ee" path="/var/lib/kubelet/pods/ecac25ea-d89b-4671-b83b-a7be7ef491ee/volumes"
Jul 28 13:49:29.418803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:29.418680    2558 generic.go:356] "Generic (PLEG): container finished" podID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerID="d69a2abea6923f9bdb4b2a9c422cc3a2f8cba3f5de7da63ee7e1afeae8b1e210" exitCode=0
Jul 28 13:49:29.418803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:29.418762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerDied","Data":"d69a2abea6923f9bdb4b2a9c422cc3a2f8cba3f5de7da63ee7e1afeae8b1e210"}
Jul 28 13:49:29.420094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:29.420080    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:49:33.441390 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:33.441357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerStarted","Data":"bd53711d4b75e6e0c9083cdcc80c17cfc779bf8d409f6c8f0b7932b0572cbeb8"}
Jul 28 13:49:33.441803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:33.441413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerStarted","Data":"7947bfae1732e0490d454d994623087326267be2c0b2e421fc020f353ae5f204"}
Jul 28 13:49:33.441803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:33.441669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:33.463667 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:33.463564    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podStartSLOduration=1.244153504 podStartE2EDuration="9.463551252s" totalImagesPullingTime="3.775014566s" totalInitContainerRuntime="4.444383182s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:49:33.462183054 +0000 UTC m=+4479.947807661" watchObservedRunningTime="2026-07-28 13:49:33.463551252 +0000 UTC m=+4479.949175843"
Jul 28 13:49:34.445769 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:34.445711    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:34.447057 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:34.447026    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 13:49:35.450623 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:35.450583    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 13:49:40.456041 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:40.456008    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:40.456644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:40.456611    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 13:49:50.456994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:50.456961    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:49:55.227487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:55.227375    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:49:55.248168 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:49:55.248143    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:50:04.645664 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:04.645628    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:50:04.646094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:04.645885    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container" containerID="cri-o://7947bfae1732e0490d454d994623087326267be2c0b2e421fc020f353ae5f204" gracePeriod=30
Jul 28 13:50:04.646094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:04.645928    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" containerID="cri-o://bd53711d4b75e6e0c9083cdcc80c17cfc779bf8d409f6c8f0b7932b0572cbeb8" gracePeriod=30
Jul 28 13:50:05.450922 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:05.450870    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:05.574288 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:05.574250    2558 generic.go:356] "Generic (PLEG): container finished" podID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerID="bd53711d4b75e6e0c9083cdcc80c17cfc779bf8d409f6c8f0b7932b0572cbeb8" exitCode=2
Jul 28 13:50:05.574451 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:05.574323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerDied","Data":"bd53711d4b75e6e0c9083cdcc80c17cfc779bf8d409f6c8f0b7932b0572cbeb8"}
Jul 28 13:50:09.836775 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.836740    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:50:09.839053 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.839032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:09.841743 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.841704    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 13:50:09.841871 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.841746    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:50:09.855907 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.855859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:50:09.968984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.968949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:09.968984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.968990    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:09.969216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.969033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:09.969216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.969122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:09.969216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:09.969176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c972n\" (UniqueName: \"kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.069586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.069547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.069586 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.069591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c972n\" (UniqueName: \"kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.069859 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.069643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.069859 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:50:10.069750    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 13:50:10.069859 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:50:10.069817    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls podName:8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f nodeName:}" failed. No retries permitted until 2026-07-28 13:50:10.569797094 +0000 UTC m=+4517.055421682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 13:50:10.070034 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.069863    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.070034 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.069948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.070303 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.070282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.070385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.070370    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.070438 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.070422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.079197 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.079174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c972n\" (UniqueName: \"kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.451705 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.451657    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:10.574369 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.574327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.576923 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.576897    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-mskcd\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.749770 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.749644    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:10.879542 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:10.879518    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:50:10.881677 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:50:10.881650    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfdf9e5_37ab_4dd6_a2b9_7976c2fdef4f.slice/crio-bd58c3b66622fc74465dbae7944250af8372a10926e454001dcd4a8ba763161c WatchSource:0}: Error finding container bd58c3b66622fc74465dbae7944250af8372a10926e454001dcd4a8ba763161c: Status 404 returned error can't find the container with id bd58c3b66622fc74465dbae7944250af8372a10926e454001dcd4a8ba763161c
Jul 28 13:50:11.600343 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:11.600309    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerStarted","Data":"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"}
Jul 28 13:50:11.600556 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:11.600366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerStarted","Data":"bd58c3b66622fc74465dbae7944250af8372a10926e454001dcd4a8ba763161c"}
Jul 28 13:50:15.451876 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:15.451780    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:15.452223 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:15.451914    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:50:15.620552 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:15.620518    2558 generic.go:356] "Generic (PLEG): container finished" podID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerID="c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f" exitCode=0
Jul 28 13:50:15.620754 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:15.620594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerDied","Data":"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"}
Jul 28 13:50:16.627114 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:16.627077    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerStarted","Data":"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"}
Jul 28 13:50:16.627530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:16.627145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerStarted","Data":"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"}
Jul 28 13:50:16.627530 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:16.627345    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:16.648061 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:16.648005    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podStartSLOduration=3.407904006 podStartE2EDuration="7.647989067s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.240085061s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:50:16.645773083 +0000 UTC m=+4523.131397674" watchObservedRunningTime="2026-07-28 13:50:16.647989067 +0000 UTC m=+4523.133613756"
Jul 28 13:50:17.631868 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:17.631834    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:17.633270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:17.633239    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 13:50:18.635178 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:18.635139    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 13:50:20.451684 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:20.451646    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:23.640175 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:23.640147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:23.640642 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:23.640615    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 13:50:25.451771 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:25.451696    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:30.451452 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:30.451405    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 28 13:50:33.640785 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:33.640748    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:50:34.702142 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:34.702113    2558 generic.go:356] "Generic (PLEG): container finished" podID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerID="7947bfae1732e0490d454d994623087326267be2c0b2e421fc020f353ae5f204" exitCode=137
Jul 28 13:50:34.702449 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:34.702160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerDied","Data":"7947bfae1732e0490d454d994623087326267be2c0b2e421fc020f353ae5f204"}
Jul 28 13:50:35.295713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.295688    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:50:35.377242 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5llbw\" (UniqueName: \"kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw\") pod \"02e0fe96-db34-43c0-90d8-8192a1088a97\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") "
Jul 28 13:50:35.377434 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config\") pod \"02e0fe96-db34-43c0-90d8-8192a1088a97\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") "
Jul 28 13:50:35.377434 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls\") pod \"02e0fe96-db34-43c0-90d8-8192a1088a97\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") "
Jul 28 13:50:35.377434 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377353    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location\") pod \"02e0fe96-db34-43c0-90d8-8192a1088a97\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") "
Jul 28 13:50:35.377434 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert\") pod \"02e0fe96-db34-43c0-90d8-8192a1088a97\" (UID: \"02e0fe96-db34-43c0-90d8-8192a1088a97\") "
Jul 28 13:50:35.377742 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377698    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02e0fe96-db34-43c0-90d8-8192a1088a97" (UID: "02e0fe96-db34-43c0-90d8-8192a1088a97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:50:35.377808 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.377777    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02e0fe96-db34-43c0-90d8-8192a1088a97" (UID: "02e0fe96-db34-43c0-90d8-8192a1088a97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:50:35.379401 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.379379    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw" (OuterVolumeSpecName: "kube-api-access-5llbw") pod "02e0fe96-db34-43c0-90d8-8192a1088a97" (UID: "02e0fe96-db34-43c0-90d8-8192a1088a97"). InnerVolumeSpecName "kube-api-access-5llbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:50:35.379713 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.379692    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02e0fe96-db34-43c0-90d8-8192a1088a97" (UID: "02e0fe96-db34-43c0-90d8-8192a1088a97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:50:35.389315 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.389280    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02e0fe96-db34-43c0-90d8-8192a1088a97" (UID: "02e0fe96-db34-43c0-90d8-8192a1088a97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:50:35.478134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.478049    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e0fe96-db34-43c0-90d8-8192a1088a97-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:50:35.478134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.478081    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:50:35.478134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.478091    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llbw\" (UniqueName: \"kubernetes.io/projected/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-api-access-5llbw\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:50:35.478134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.478099    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e0fe96-db34-43c0-90d8-8192a1088a97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:50:35.478134 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.478110    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e0fe96-db34-43c0-90d8-8192a1088a97-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:50:35.708587 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.708554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj" event={"ID":"02e0fe96-db34-43c0-90d8-8192a1088a97","Type":"ContainerDied","Data":"365cafdfb05f6f10c89d1b5fc3e72ab8449359028ff9171c0ef5001669e57c01"}
Jul 28 13:50:35.709087 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.708616    2558 scope.go:117] "RemoveContainer" containerID="bd53711d4b75e6e0c9083cdcc80c17cfc779bf8d409f6c8f0b7932b0572cbeb8"
Jul 28 13:50:35.709087 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.708561    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"
Jul 28 13:50:35.717605 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.717587    2558 scope.go:117] "RemoveContainer" containerID="7947bfae1732e0490d454d994623087326267be2c0b2e421fc020f353ae5f204"
Jul 28 13:50:35.725331 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.725314    2558 scope.go:117] "RemoveContainer" containerID="d69a2abea6923f9bdb4b2a9c422cc3a2f8cba3f5de7da63ee7e1afeae8b1e210"
Jul 28 13:50:35.730741 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.730658    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:50:35.736256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:35.736232    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9jqj"]
Jul 28 13:50:36.017638 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:36.017545    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" path="/var/lib/kubelet/pods/02e0fe96-db34-43c0-90d8-8192a1088a97/volumes"
Jul 28 13:50:50.226850 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:50.226791    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:50:50.227372 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:50.227113    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container" containerID="cri-o://579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2" gracePeriod=30
Jul 28 13:50:50.227372 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:50.227181    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" containerID="cri-o://b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9" gracePeriod=30
Jul 28 13:50:50.770763 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:50.770707    2558 generic.go:356] "Generic (PLEG): container finished" podID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerID="b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9" exitCode=2
Jul 28 13:50:50.770913 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:50.770779    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerDied","Data":"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"}
Jul 28 13:50:53.635494 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:53.635454    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:50:55.418339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418297    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:50:55.418826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418787    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container"
Jul 28 13:50:55.418826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418803    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container"
Jul 28 13:50:55.418826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418825    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="storage-initializer"
Jul 28 13:50:55.418953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418834    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="storage-initializer"
Jul 28 13:50:55.418953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418858    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy"
Jul 28 13:50:55.418953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418866    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy"
Jul 28 13:50:55.418953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418934    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kube-rbac-proxy"
Jul 28 13:50:55.418953 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.418948    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e0fe96-db34-43c0-90d8-8192a1088a97" containerName="kserve-container"
Jul 28 13:50:55.423909 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.423885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.426467 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.426450    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 13:50:55.426558 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.426450    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 13:50:55.431596 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.431568    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:50:55.541025 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.540985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dph\" (UniqueName: \"kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.541211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.541032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.541211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.541065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.541211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.541085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.541211 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.541108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.641965 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.641927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642157 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dph\" (UniqueName: \"kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642157 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642157 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642338 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642467 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642767 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.642848 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.642766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.644585 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.644568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.650905 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.650881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dph\" (UniqueName: \"kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph\") pod \"isvc-triton-predictor-5d7db58676-ltsnb\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.737037 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.736947    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:50:55.872102 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:55.872074    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:50:55.874036 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:50:55.874013    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba1c2c9_38ee_4463_9c80_61bfd9873fa6.slice/crio-f65d5db76baa34c368a212c531ee652516d558e4fd5690e11a60524eba8a13e6 WatchSource:0}: Error finding container f65d5db76baa34c368a212c531ee652516d558e4fd5690e11a60524eba8a13e6: Status 404 returned error can't find the container with id f65d5db76baa34c368a212c531ee652516d558e4fd5690e11a60524eba8a13e6
Jul 28 13:50:56.796313 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:56.796276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerStarted","Data":"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"}
Jul 28 13:50:56.796712 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:56.796333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerStarted","Data":"f65d5db76baa34c368a212c531ee652516d558e4fd5690e11a60524eba8a13e6"}
Jul 28 13:50:58.636058 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:58.636012    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:50:59.809590 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:59.809557    2558 generic.go:356] "Generic (PLEG): container finished" podID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerID="949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6" exitCode=0
Jul 28 13:50:59.809996 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:50:59.809634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerDied","Data":"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"}
Jul 28 13:51:03.635583 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:03.635535    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:51:03.636285 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:03.635685    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:51:08.636006 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:08.635463    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:51:13.635695 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:13.635649    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:51:18.636334 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:18.636213    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 13:51:20.947838 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.947794    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:51:20.963186 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.963152    2558 generic.go:356] "Generic (PLEG): container finished" podID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerID="579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2" exitCode=137
Jul 28 13:51:20.963354 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.963207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerDied","Data":"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"}
Jul 28 13:51:20.963354 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.963295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd" event={"ID":"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f","Type":"ContainerDied","Data":"bd58c3b66622fc74465dbae7944250af8372a10926e454001dcd4a8ba763161c"}
Jul 28 13:51:20.963354 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.963330    2558 scope.go:117] "RemoveContainer" containerID="b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"
Jul 28 13:51:20.963477 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.963427    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"
Jul 28 13:51:20.974336 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.974307    2558 scope.go:117] "RemoveContainer" containerID="579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"
Jul 28 13:51:20.986253 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.986169    2558 scope.go:117] "RemoveContainer" containerID="c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"
Jul 28 13:51:20.998482 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.998432    2558 scope.go:117] "RemoveContainer" containerID="b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"
Jul 28 13:51:20.998832 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:51:20.998795    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9\": container with ID starting with b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9 not found: ID does not exist" containerID="b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"
Jul 28 13:51:20.998933 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.998835    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9"} err="failed to get container status \"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9\": rpc error: code = NotFound desc = could not find container \"b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9\": container with ID starting with b40fd0ac05c9434268fe8b3774b9e8344d6eb4b1b571c8a28df7e200576862e9 not found: ID does not exist"
Jul 28 13:51:20.998933 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.998854    2558 scope.go:117] "RemoveContainer" containerID="579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"
Jul 28 13:51:20.999246 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:51:20.999217    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2\": container with ID starting with 579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2 not found: ID does not exist" containerID="579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"
Jul 28 13:51:20.999321 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.999254    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2"} err="failed to get container status \"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2\": rpc error: code = NotFound desc = could not find container \"579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2\": container with ID starting with 579235cddb05f694a1acce0ebc5a983f63dc07bd77ba8663b0391b889d4f1ec2 not found: ID does not exist"
Jul 28 13:51:20.999321 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.999278    2558 scope.go:117] "RemoveContainer" containerID="c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"
Jul 28 13:51:20.999673 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:51:20.999651    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f\": container with ID starting with c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f not found: ID does not exist" containerID="c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"
Jul 28 13:51:20.999799 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:20.999679    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f"} err="failed to get container status \"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f\": rpc error: code = NotFound desc = could not find container \"c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f\": container with ID starting with c9d7565613b2a018a95b395c566ee5467c0834bdd3e7bce11915167adf99085f not found: ID does not exist"
Jul 28 13:51:21.073368 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") pod \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") "
Jul 28 13:51:21.073368 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073340    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config\") pod \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") "
Jul 28 13:51:21.073595 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073423    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert\") pod \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") "
Jul 28 13:51:21.073595 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location\") pod \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") "
Jul 28 13:51:21.073595 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073505    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c972n\" (UniqueName: \"kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n\") pod \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\" (UID: \"8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f\") "
Jul 28 13:51:21.073899 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073788    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:21.073981 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.073902    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:21.076122 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.076085    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:51:21.076513 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.076489    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n" (OuterVolumeSpecName: "kube-api-access-c972n") pod "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f"). InnerVolumeSpecName "kube-api-access-c972n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:51:21.086063 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.086032    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" (UID: "8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:51:21.174636 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.174595    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:51:21.174636 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.174637    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:51:21.174917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.174653    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:51:21.174917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.174664    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:51:21.174917 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.174673    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c972n\" (UniqueName: \"kubernetes.io/projected/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f-kube-api-access-c972n\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:51:21.291963 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.291908    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:51:21.298575 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:21.298547    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-mskcd"]
Jul 28 13:51:22.019155 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:51:22.018894    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" path="/var/lib/kubelet/pods/8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f/volumes"
Jul 28 13:52:53.406355 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:53.406322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerStarted","Data":"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"}
Jul 28 13:52:53.406896 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:53.406378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerStarted","Data":"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"}
Jul 28 13:52:53.406896 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:53.406470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:52:53.434127 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:53.434074    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" podStartSLOduration=1.7809727340000001 podStartE2EDuration="1m58.434052491s" totalImagesPullingTime="1m52.871614939s" totalInitContainerRuntime="3.781464818s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:52:53.432767476 +0000 UTC m=+4679.918392067" watchObservedRunningTime="2026-07-28 13:52:53.434052491 +0000 UTC m=+4679.919677082"
Jul 28 13:52:54.410984 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:54.410945    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:52:54.412367 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:54.412336    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 13:52:55.415217 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:52:55.415177    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 13:53:00.419345 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:00.419315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:53:00.420057 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:00.420037    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:53:08.046458 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:08.046419    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:53:08.046966 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:08.046653    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container" containerID="cri-o://c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193" gracePeriod=30
Jul 28 13:53:08.046966 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:08.046699    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kube-rbac-proxy" containerID="cri-o://fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1" gracePeriod=30
Jul 28 13:53:08.469696 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:08.469659    2558 generic.go:356] "Generic (PLEG): container finished" podID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerID="fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1" exitCode=2
Jul 28 13:53:08.469892 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:08.469708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerDied","Data":"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"}
Jul 28 13:53:10.389487 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.389463    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:53:10.484861 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.484826    2558 generic.go:356] "Generic (PLEG): container finished" podID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerID="c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193" exitCode=0
Jul 28 13:53:10.485030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.484873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerDied","Data":"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"}
Jul 28 13:53:10.485030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.484913    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"
Jul 28 13:53:10.485030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.484941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb" event={"ID":"dba1c2c9-38ee-4463-9c80-61bfd9873fa6","Type":"ContainerDied","Data":"f65d5db76baa34c368a212c531ee652516d558e4fd5690e11a60524eba8a13e6"}
Jul 28 13:53:10.485030 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.484967    2558 scope.go:117] "RemoveContainer" containerID="fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"
Jul 28 13:53:10.493250 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.493233    2558 scope.go:117] "RemoveContainer" containerID="c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"
Jul 28 13:53:10.500703 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.500686    2558 scope.go:117] "RemoveContainer" containerID="949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"
Jul 28 13:53:10.507782 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.507752    2558 scope.go:117] "RemoveContainer" containerID="fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"
Jul 28 13:53:10.508050 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:53:10.508028    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1\": container with ID starting with fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1 not found: ID does not exist" containerID="fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"
Jul 28 13:53:10.508108 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.508060    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1"} err="failed to get container status \"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1\": rpc error: code = NotFound desc = could not find container \"fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1\": container with ID starting with fa5d3d06c5d918a8330c5a415bff6f1360bf58982c271406e5da679224ba36e1 not found: ID does not exist"
Jul 28 13:53:10.508108 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.508078    2558 scope.go:117] "RemoveContainer" containerID="c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"
Jul 28 13:53:10.508315 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:53:10.508298    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193\": container with ID starting with c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193 not found: ID does not exist" containerID="c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"
Jul 28 13:53:10.508364 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.508321    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193"} err="failed to get container status \"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193\": rpc error: code = NotFound desc = could not find container \"c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193\": container with ID starting with c93951c2f851c6cbfc76ee78086cc9375588f7bad8b5dcfc36c0e1e3d6806193 not found: ID does not exist"
Jul 28 13:53:10.508364 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.508335    2558 scope.go:117] "RemoveContainer" containerID="949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"
Jul 28 13:53:10.508521 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:53:10.508506    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6\": container with ID starting with 949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6 not found: ID does not exist" containerID="949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"
Jul 28 13:53:10.508559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.508524    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6"} err="failed to get container status \"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6\": rpc error: code = NotFound desc = could not find container \"949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6\": container with ID starting with 949559ecb2e7186058884b0139ff60602b1c5dfe74da48f887415955fa26e2e6 not found: ID does not exist"
Jul 28 13:53:10.519836 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.519818    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert\") pod \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") "
Jul 28 13:53:10.519920 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.519861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location\") pod \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") "
Jul 28 13:53:10.519920 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.519892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls\") pod \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") "
Jul 28 13:53:10.520033 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.519930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config\") pod \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") "
Jul 28 13:53:10.520033 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.519970    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9dph\" (UniqueName: \"kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph\") pod \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\" (UID: \"dba1c2c9-38ee-4463-9c80-61bfd9873fa6\") "
Jul 28 13:53:10.520256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.520233    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dba1c2c9-38ee-4463-9c80-61bfd9873fa6" (UID: "dba1c2c9-38ee-4463-9c80-61bfd9873fa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:53:10.520256 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.520235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dba1c2c9-38ee-4463-9c80-61bfd9873fa6" (UID: "dba1c2c9-38ee-4463-9c80-61bfd9873fa6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:53:10.520382 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.520262    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dba1c2c9-38ee-4463-9c80-61bfd9873fa6" (UID: "dba1c2c9-38ee-4463-9c80-61bfd9873fa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:53:10.522010 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.521993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dba1c2c9-38ee-4463-9c80-61bfd9873fa6" (UID: "dba1c2c9-38ee-4463-9c80-61bfd9873fa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:53:10.522229 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.522209    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph" (OuterVolumeSpecName: "kube-api-access-s9dph") pod "dba1c2c9-38ee-4463-9c80-61bfd9873fa6" (UID: "dba1c2c9-38ee-4463-9c80-61bfd9873fa6"). InnerVolumeSpecName "kube-api-access-s9dph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:53:10.620524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.620488    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:53:10.620524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.620516    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dph\" (UniqueName: \"kubernetes.io/projected/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kube-api-access-s9dph\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:53:10.620524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.620525    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:53:10.620524 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.620535    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:53:10.620821 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.620544    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba1c2c9-38ee-4463-9c80-61bfd9873fa6-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:53:10.808661 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.808629    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:53:10.812497 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:10.812472    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-ltsnb"]
Jul 28 13:53:12.017490 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:12.017458    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" path="/var/lib/kubelet/pods/dba1c2c9-38ee-4463-9c80-61bfd9873fa6/volumes"
Jul 28 13:53:13.266532 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266494    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266810    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="storage-initializer"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266821    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="storage-initializer"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266829    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266834    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266841    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kube-rbac-proxy"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266847    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kube-rbac-proxy"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266857    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="storage-initializer"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266864    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="storage-initializer"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266874    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266879    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266891    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266896    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266948    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266956    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kserve-container"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266964    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba1c2c9-38ee-4463-9c80-61bfd9873fa6" containerName="kube-rbac-proxy"
Jul 28 13:53:13.266985 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.266969    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfdf9e5-37ab-4dd6-a2b9-7976c2fdef4f" containerName="kube-rbac-proxy"
Jul 28 13:53:13.296299 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.296267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.299285 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.299248    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:53:13.299285 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.299261    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 13:53:13.299285 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.299277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:53:13.299547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.299395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:53:13.299547 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.299534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:53:13.300428 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.300411    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:53:13.300523 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.300440    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 13:53:13.308379 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.308358    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:53:13.442676 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.442636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.442676 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.442674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.442911 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.442696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.442911 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.442812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggmk\" (UniqueName: \"kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.442911 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.442845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.543924 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.543827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggmk\" (UniqueName: \"kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.543924 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.543874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544145 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.543952    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544145 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.544068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544145 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.544103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544404 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.544386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544664 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.544648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.544746 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.544709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.546533 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.546510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.553456 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.553438    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggmk\" (UniqueName: \"kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk\") pod \"isvc-xgboost-predictor-7566d65bf7-2x7bm\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.607796 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.607754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:13.744109 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:13.744045    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:53:13.746826 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:53:13.746797    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7489975_6c51_405d_91a7_32b5248fe262.slice/crio-71bb5337fed83aa4aa6f73282f20f8491f6f9ef6cf17f966d75ea7d8df5b0274 WatchSource:0}: Error finding container 71bb5337fed83aa4aa6f73282f20f8491f6f9ef6cf17f966d75ea7d8df5b0274: Status 404 returned error can't find the container with id 71bb5337fed83aa4aa6f73282f20f8491f6f9ef6cf17f966d75ea7d8df5b0274
Jul 28 13:53:14.503047 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:14.503007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerStarted","Data":"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"}
Jul 28 13:53:14.503431 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:14.503078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerStarted","Data":"71bb5337fed83aa4aa6f73282f20f8491f6f9ef6cf17f966d75ea7d8df5b0274"}
Jul 28 13:53:18.525918 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:18.525884    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7489975-6c51-405d-91a7-32b5248fe262" containerID="f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc" exitCode=0
Jul 28 13:53:18.526297 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:18.525950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerDied","Data":"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"}
Jul 28 13:53:36.610781 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:36.610745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerStarted","Data":"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"}
Jul 28 13:53:36.611194 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:36.610810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerStarted","Data":"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"}
Jul 28 13:53:36.611194 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:36.610988    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:36.637092 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:36.637040    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podStartSLOduration=2.266281863 podStartE2EDuration="23.637024574s" totalImagesPullingTime="17.320471761s" totalInitContainerRuntime="4.05027095s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:53:36.634306782 +0000 UTC m=+4723.119931375" watchObservedRunningTime="2026-07-28 13:53:36.637024574 +0000 UTC m=+4723.122649165"
Jul 28 13:53:37.615162 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:37.615126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:37.616709 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:37.616681    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:53:38.619035 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:38.618998    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:53:43.623017 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:43.622987    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:53:43.623479 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:43.623452    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:53:53.623803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:53:53.623757    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:54:03.623508 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:03.623465    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:54:13.623897 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:13.623854    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:54:23.623575 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:23.623534    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:54:33.623880 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:33.623849    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:54:43.330738 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.330678    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:54:43.331287 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.330992    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" containerID="cri-o://0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847" gracePeriod=30
Jul 28 13:54:43.331287 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.331049    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kube-rbac-proxy" containerID="cri-o://bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87" gracePeriod=30
Jul 28 13:54:43.619853 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.619813    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 28 13:54:43.624104 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.624081    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 13:54:43.887132 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.887047    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7489975-6c51-405d-91a7-32b5248fe262" containerID="bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87" exitCode=2
Jul 28 13:54:43.887132 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:43.887098    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerDied","Data":"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"}
Jul 28 13:54:47.078556 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.078527    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:54:47.125683 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.125650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tggmk\" (UniqueName: \"kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk\") pod \"e7489975-6c51-405d-91a7-32b5248fe262\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") "
Jul 28 13:54:47.125887 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.125691    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls\") pod \"e7489975-6c51-405d-91a7-32b5248fe262\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") "
Jul 28 13:54:47.125887 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.125762    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert\") pod \"e7489975-6c51-405d-91a7-32b5248fe262\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") "
Jul 28 13:54:47.125887 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.125833    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location\") pod \"e7489975-6c51-405d-91a7-32b5248fe262\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") "
Jul 28 13:54:47.125887 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.125859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config\") pod \"e7489975-6c51-405d-91a7-32b5248fe262\" (UID: \"e7489975-6c51-405d-91a7-32b5248fe262\") "
Jul 28 13:54:47.126212 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.126174    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7489975-6c51-405d-91a7-32b5248fe262" (UID: "e7489975-6c51-405d-91a7-32b5248fe262"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:54:47.126212 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.126202    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7489975-6c51-405d-91a7-32b5248fe262" (UID: "e7489975-6c51-405d-91a7-32b5248fe262"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:54:47.126528 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.126250    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7489975-6c51-405d-91a7-32b5248fe262" (UID: "e7489975-6c51-405d-91a7-32b5248fe262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:54:47.127991 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.127963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk" (OuterVolumeSpecName: "kube-api-access-tggmk") pod "e7489975-6c51-405d-91a7-32b5248fe262" (UID: "e7489975-6c51-405d-91a7-32b5248fe262"). InnerVolumeSpecName "kube-api-access-tggmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:54:47.127991 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.127979    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7489975-6c51-405d-91a7-32b5248fe262" (UID: "e7489975-6c51-405d-91a7-32b5248fe262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:54:47.226841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.226760    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7489975-6c51-405d-91a7-32b5248fe262-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:54:47.226841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.226787    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:54:47.226841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.226796    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tggmk\" (UniqueName: \"kubernetes.io/projected/e7489975-6c51-405d-91a7-32b5248fe262-kube-api-access-tggmk\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:54:47.226841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.226804    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7489975-6c51-405d-91a7-32b5248fe262-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:54:47.226841 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.226814    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7489975-6c51-405d-91a7-32b5248fe262-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:54:47.909367 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.909338    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7489975-6c51-405d-91a7-32b5248fe262" containerID="0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847" exitCode=0
Jul 28 13:54:47.909559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.909406    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"
Jul 28 13:54:47.909559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.909427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerDied","Data":"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"}
Jul 28 13:54:47.909559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.909491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm" event={"ID":"e7489975-6c51-405d-91a7-32b5248fe262","Type":"ContainerDied","Data":"71bb5337fed83aa4aa6f73282f20f8491f6f9ef6cf17f966d75ea7d8df5b0274"}
Jul 28 13:54:47.909559 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.909519    2558 scope.go:117] "RemoveContainer" containerID="bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"
Jul 28 13:54:47.918694 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.918675    2558 scope.go:117] "RemoveContainer" containerID="0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"
Jul 28 13:54:47.926094 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.926078    2558 scope.go:117] "RemoveContainer" containerID="f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"
Jul 28 13:54:47.932071 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.932048    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:54:47.933315 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.933297    2558 scope.go:117] "RemoveContainer" containerID="bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"
Jul 28 13:54:47.933563 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:54:47.933545    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87\": container with ID starting with bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87 not found: ID does not exist" containerID="bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"
Jul 28 13:54:47.933611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.933572    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87"} err="failed to get container status \"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87\": rpc error: code = NotFound desc = could not find container \"bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87\": container with ID starting with bf5a35714266a57e70cdfee90f1a5d48a2a7ec3d4a4ea4339a08136811203e87 not found: ID does not exist"
Jul 28 13:54:47.933611 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.933588    2558 scope.go:117] "RemoveContainer" containerID="0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"
Jul 28 13:54:47.933847 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:54:47.933830    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847\": container with ID starting with 0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847 not found: ID does not exist" containerID="0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"
Jul 28 13:54:47.933891 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.933854    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847"} err="failed to get container status \"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847\": rpc error: code = NotFound desc = could not find container \"0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847\": container with ID starting with 0f1e7d31b0fc7e768c9b584ef08f689a1950cb19e8805e80e159c36fade7f847 not found: ID does not exist"
Jul 28 13:54:47.933891 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.933870    2558 scope.go:117] "RemoveContainer" containerID="f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"
Jul 28 13:54:47.934126 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:54:47.934081    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc\": container with ID starting with f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc not found: ID does not exist" containerID="f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"
Jul 28 13:54:47.934126 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.934107    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc"} err="failed to get container status \"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc\": rpc error: code = NotFound desc = could not find container \"f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc\": container with ID starting with f7f63b33615a9e451609f2171cdbdb192023cdd7058abc2c06347a502e193cfc not found: ID does not exist"
Jul 28 13:54:47.936370 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:47.936352    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2x7bm"]
Jul 28 13:54:48.018340 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:48.018307    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7489975-6c51-405d-91a7-32b5248fe262" path="/var/lib/kubelet/pods/e7489975-6c51-405d-91a7-32b5248fe262/volumes"
Jul 28 13:54:55.248855 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:55.248753    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:54:55.270766 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:54:55.270742    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:56:54.168669 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168633    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168949    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="storage-initializer"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168961    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="storage-initializer"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168970    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168975    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168981    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kube-rbac-proxy"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.168987    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kube-rbac-proxy"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.169039    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kserve-container"
Jul 28 13:56:54.169177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.169047    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7489975-6c51-405d-91a7-32b5248fe262" containerName="kube-rbac-proxy"
Jul 28 13:56:54.170943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.170926    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.173830 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.173806    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:56:54.175923 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.175908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:56:54.176272 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.176253    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:56:54.176394 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.176287    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 13:56:54.176564 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.176545    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:56:54.177185 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.177172    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:56:54.177272 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.177197    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:56:54.183632 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.183614    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:56:54.333628 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.333595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.333826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.333638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.333826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.333661    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8j6\" (UniqueName: \"kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.333826 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.333809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.333940 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.333849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.434616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.434511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.434616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.434570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.434616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.434601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.434616 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.434620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8j6\" (UniqueName: \"kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.435028 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.434672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.435119 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.435095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.435245 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.435226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.435327 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.435311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.437267 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.437241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.443558 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.443537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8j6\" (UniqueName: \"kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2qcst\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.483236 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.483202    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:54.615766 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:54.615705    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:56:54.618211 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:56:54.618183    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f45480_31d2_469b_a693_8ea01da2a34f.slice/crio-91a23f93f83561cd6d851dee10e3f519cfee25d13a47c9f85fe8af635ccc3692 WatchSource:0}: Error finding container 91a23f93f83561cd6d851dee10e3f519cfee25d13a47c9f85fe8af635ccc3692: Status 404 returned error can't find the container with id 91a23f93f83561cd6d851dee10e3f519cfee25d13a47c9f85fe8af635ccc3692
Jul 28 13:56:55.413156 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:55.413109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerStarted","Data":"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"}
Jul 28 13:56:55.413575 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:55.413180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerStarted","Data":"91a23f93f83561cd6d851dee10e3f519cfee25d13a47c9f85fe8af635ccc3692"}
Jul 28 13:56:58.427058 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:58.427026    2558 generic.go:356] "Generic (PLEG): container finished" podID="39f45480-31d2-469b-a693-8ea01da2a34f" containerID="1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39" exitCode=0
Jul 28 13:56:58.427429 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:58.427105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerDied","Data":"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"}
Jul 28 13:56:59.435994 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.435955    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerStarted","Data":"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"}
Jul 28 13:56:59.436427 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.436030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerStarted","Data":"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"}
Jul 28 13:56:59.436427 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.436316    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:59.436538 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.436466    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:56:59.437691 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.437663    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:56:59.458101 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:56:59.458051    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podStartSLOduration=1.78247896 podStartE2EDuration="5.458036557s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675557597s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:56:59.45684709 +0000 UTC m=+4925.942471681" watchObservedRunningTime="2026-07-28 13:56:59.458036557 +0000 UTC m=+4925.943661224"
Jul 28 13:57:00.440787 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:00.440744    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:05.445315 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:05.445285    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:57:05.445894 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:05.445869    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:15.446509 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:15.446469    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:25.445972 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:25.445933    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:35.446413 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:35.446375    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:45.446708 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:45.446663    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:57:55.446469 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:57:55.446436    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:58:04.282938 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:04.282904    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:58:04.283348 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:04.283183    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" containerID="cri-o://c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01" gracePeriod=30
Jul 28 13:58:04.283348 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:04.283223    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kube-rbac-proxy" containerID="cri-o://e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2" gracePeriod=30
Jul 28 13:58:04.703923 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:04.703889    2558 generic.go:356] "Generic (PLEG): container finished" podID="39f45480-31d2-469b-a693-8ea01da2a34f" containerID="e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2" exitCode=2
Jul 28 13:58:04.704095 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:04.703960    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerDied","Data":"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"}
Jul 28 13:58:05.441683 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:05.441638    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 28 13:58:05.446010 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:05.445981    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 13:58:08.036639 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.036614    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:58:08.129943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.129909    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location\") pod \"39f45480-31d2-469b-a693-8ea01da2a34f\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") "
Jul 28 13:58:08.129943 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.129953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config\") pod \"39f45480-31d2-469b-a693-8ea01da2a34f\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") "
Jul 28 13:58:08.130151 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.129979    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls\") pod \"39f45480-31d2-469b-a693-8ea01da2a34f\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") "
Jul 28 13:58:08.130151 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.129996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert\") pod \"39f45480-31d2-469b-a693-8ea01da2a34f\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") "
Jul 28 13:58:08.130151 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.130024    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg8j6\" (UniqueName: \"kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6\") pod \"39f45480-31d2-469b-a693-8ea01da2a34f\" (UID: \"39f45480-31d2-469b-a693-8ea01da2a34f\") "
Jul 28 13:58:08.130318 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.130269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39f45480-31d2-469b-a693-8ea01da2a34f" (UID: "39f45480-31d2-469b-a693-8ea01da2a34f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:58:08.130383 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.130361    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39f45480-31d2-469b-a693-8ea01da2a34f" (UID: "39f45480-31d2-469b-a693-8ea01da2a34f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:58:08.130440 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.130382    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39f45480-31d2-469b-a693-8ea01da2a34f" (UID: "39f45480-31d2-469b-a693-8ea01da2a34f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:58:08.132216 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.132190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6" (OuterVolumeSpecName: "kube-api-access-dg8j6") pod "39f45480-31d2-469b-a693-8ea01da2a34f" (UID: "39f45480-31d2-469b-a693-8ea01da2a34f"). InnerVolumeSpecName "kube-api-access-dg8j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:58:08.132352 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.132259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39f45480-31d2-469b-a693-8ea01da2a34f" (UID: "39f45480-31d2-469b-a693-8ea01da2a34f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:58:08.230644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.230612    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:58:08.230644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.230641    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f45480-31d2-469b-a693-8ea01da2a34f-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:58:08.230644 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.230651    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f45480-31d2-469b-a693-8ea01da2a34f-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:58:08.230922 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.230660    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8j6\" (UniqueName: \"kubernetes.io/projected/39f45480-31d2-469b-a693-8ea01da2a34f-kube-api-access-dg8j6\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:58:08.230922 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.230668    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f45480-31d2-469b-a693-8ea01da2a34f-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 13:58:08.722854 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.722817    2558 generic.go:356] "Generic (PLEG): container finished" podID="39f45480-31d2-469b-a693-8ea01da2a34f" containerID="c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01" exitCode=0
Jul 28 13:58:08.723073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.722875    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"
Jul 28 13:58:08.723073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.722906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerDied","Data":"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"}
Jul 28 13:58:08.723073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.722982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst" event={"ID":"39f45480-31d2-469b-a693-8ea01da2a34f","Type":"ContainerDied","Data":"91a23f93f83561cd6d851dee10e3f519cfee25d13a47c9f85fe8af635ccc3692"}
Jul 28 13:58:08.723073 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.723020    2558 scope.go:117] "RemoveContainer" containerID="e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"
Jul 28 13:58:08.731555 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.731538    2558 scope.go:117] "RemoveContainer" containerID="c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"
Jul 28 13:58:08.738876 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.738859    2558 scope.go:117] "RemoveContainer" containerID="1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"
Jul 28 13:58:08.743803 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.743687    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:58:08.747398 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.747370    2558 scope.go:117] "RemoveContainer" containerID="e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"
Jul 28 13:58:08.747630 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.747613    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2qcst"]
Jul 28 13:58:08.747693 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:58:08.747664    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2\": container with ID starting with e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2 not found: ID does not exist" containerID="e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"
Jul 28 13:58:08.747761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.747687    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2"} err="failed to get container status \"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2\": rpc error: code = NotFound desc = could not find container \"e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2\": container with ID starting with e6ff1d127427f311d088062faaf97c32b762cb8b2f342431369bc169d4ca13b2 not found: ID does not exist"
Jul 28 13:58:08.747761 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.747702    2558 scope.go:117] "RemoveContainer" containerID="c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"
Jul 28 13:58:08.747992 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:58:08.747975    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01\": container with ID starting with c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01 not found: ID does not exist" containerID="c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"
Jul 28 13:58:08.748029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.747997    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01"} err="failed to get container status \"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01\": rpc error: code = NotFound desc = could not find container \"c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01\": container with ID starting with c388cfe0d7507126a6666c1b371666180f6baf378ed0db83cdec9a6d92504c01 not found: ID does not exist"
Jul 28 13:58:08.748029 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.748011    2558 scope.go:117] "RemoveContainer" containerID="1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"
Jul 28 13:58:08.748270 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:58:08.748245    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39\": container with ID starting with 1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39 not found: ID does not exist" containerID="1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"
Jul 28 13:58:08.748362 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:08.748270    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39"} err="failed to get container status \"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39\": rpc error: code = NotFound desc = could not find container \"1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39\": container with ID starting with 1212b351fb24c7bb7a1f6553f813dd42584f9a737e778de236c42ccc0dd87b39 not found: ID does not exist"
Jul 28 13:58:10.017537 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:58:10.017506    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" path="/var/lib/kubelet/pods/39f45480-31d2-469b-a693-8ea01da2a34f/volumes"
Jul 28 13:59:14.818835 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.818799    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819107    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="storage-initializer"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819118    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="storage-initializer"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819132    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kube-rbac-proxy"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819137    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kube-rbac-proxy"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819145    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819150    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819198    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kube-rbac-proxy"
Jul 28 13:59:14.819311 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.819207    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f45480-31d2-469b-a693-8ea01da2a34f" containerName="kserve-container"
Jul 28 13:59:14.822247 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.822229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.825143 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.825118    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:59:14.825287 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.825117    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 13:59:14.825287 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.825129    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:59:14.825585 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.825570    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:59:14.826561 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.826496    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:59:14.826561 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.826529    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:59:14.826751 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.826624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:59:14.840166 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.840137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 13:59:14.855047 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.855010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.855047 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.855045    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.855270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.855122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.855270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.855153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.855270 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.855172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkcd\" (UniqueName: \"kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956264 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956264 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkcd\" (UniqueName: \"kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956515 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.956515 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:59:14.956446    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 13:59:14.956701 ip-10-0-138-35 kubenswrapper[2558]: E0728 13:59:14.956539    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls podName:acc2185e-7bce-4dbf-9abe-901e9d3a0d4b nodeName:}" failed. No retries permitted until 2026-07-28 13:59:15.456518003 +0000 UTC m=+5061.942142586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-szw26" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 13:59:14.956775 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.956705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.957034 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.957014    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.957148 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.957129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:14.967677 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:14.967657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkcd\" (UniqueName: \"kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:15.459832 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:15.459792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:15.462339 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:15.462315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-szw26\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:15.734622 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:15.734530    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:15.866956 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:15.866930    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 13:59:15.868702 ip-10-0-138-35 kubenswrapper[2558]: W0728 13:59:15.868662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc2185e_7bce_4dbf_9abe_901e9d3a0d4b.slice/crio-c6012c6f1fa53d9d806e8e251a73705e182f0816c924d0e8658641cd0bdd362a WatchSource:0}: Error finding container c6012c6f1fa53d9d806e8e251a73705e182f0816c924d0e8658641cd0bdd362a: Status 404 returned error can't find the container with id c6012c6f1fa53d9d806e8e251a73705e182f0816c924d0e8658641cd0bdd362a
Jul 28 13:59:16.002931 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:16.002827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerStarted","Data":"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"}
Jul 28 13:59:16.002931 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:16.002908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerStarted","Data":"c6012c6f1fa53d9d806e8e251a73705e182f0816c924d0e8658641cd0bdd362a"}
Jul 28 13:59:20.020527 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:20.020442    2558 generic.go:356] "Generic (PLEG): container finished" podID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerID="ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7" exitCode=0
Jul 28 13:59:20.020527 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:20.020507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerDied","Data":"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"}
Jul 28 13:59:21.026806 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:21.026770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerStarted","Data":"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"}
Jul 28 13:59:21.027406 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:21.026843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerStarted","Data":"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"}
Jul 28 13:59:21.027406 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:21.027070    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:21.048385 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:21.048337    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podStartSLOduration=3.329605666 podStartE2EDuration="7.048321716s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71871605s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:59:21.046875056 +0000 UTC m=+5067.532499665" watchObservedRunningTime="2026-07-28 13:59:21.048321716 +0000 UTC m=+5067.533946308"
Jul 28 13:59:22.031176 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:22.031142    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:22.032338 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:22.032312    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 13:59:23.034790 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:23.034747    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 13:59:28.039063 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:28.039033    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 13:59:28.039585 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:28.039555    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 13:59:38.039694 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:38.039648    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 13:59:48.040091 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:48.040046    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 13:59:55.271307 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:55.271203    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:59:55.295177 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:55.295155    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 13:59:58.040115 ip-10-0-138-35 kubenswrapper[2558]: I0728 13:59:58.040073    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 14:00:08.039609 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:08.039567    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 14:00:18.040556 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:18.040476    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 14:00:24.905568 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:24.905533    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 14:00:24.906119 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:24.905760    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" containerID="cri-o://a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190" gracePeriod=30
Jul 28 14:00:24.906119 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:24.905832    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kube-rbac-proxy" containerID="cri-o://fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b" gracePeriod=30
Jul 28 14:00:25.290935 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:25.290838    2558 generic.go:356] "Generic (PLEG): container finished" podID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerID="fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b" exitCode=2
Jul 28 14:00:25.290935 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:25.290914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerDied","Data":"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"}
Jul 28 14:00:28.035846 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.035800    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 28 14:00:28.040085 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.040055    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 14:00:28.654027 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.654001    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 14:00:28.656936 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.656912    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtkcd\" (UniqueName: \"kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd\") pod \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") "
Jul 28 14:00:28.657075 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.656956    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert\") pod \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") "
Jul 28 14:00:28.657166 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657128    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config\") pod \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") "
Jul 28 14:00:28.657288 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") pod \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") "
Jul 28 14:00:28.657288 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location\") pod \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\" (UID: \"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b\") "
Jul 28 14:00:28.657288 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657255    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:00:28.657476 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657458    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:00:28.657536 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657464    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:00:28.657657 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.657637    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:00:28.659303 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.659285    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd" (OuterVolumeSpecName: "kube-api-access-jtkcd") pod "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b"). InnerVolumeSpecName "kube-api-access-jtkcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:00:28.659525 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.659502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" (UID: "acc2185e-7bce-4dbf-9abe-901e9d3a0d4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:00:28.758759 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.758635    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkcd\" (UniqueName: \"kubernetes.io/projected/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-api-access-jtkcd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:00:28.758759 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.758670    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:00:28.758759 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.758683    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:00:28.758759 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:28.758697    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:00:29.309325 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.309291    2558 generic.go:356] "Generic (PLEG): container finished" podID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerID="a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190" exitCode=0
Jul 28 14:00:29.309737 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.309352    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"
Jul 28 14:00:29.309737 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.309378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerDied","Data":"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"}
Jul 28 14:00:29.309737 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.309434    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26" event={"ID":"acc2185e-7bce-4dbf-9abe-901e9d3a0d4b","Type":"ContainerDied","Data":"c6012c6f1fa53d9d806e8e251a73705e182f0816c924d0e8658641cd0bdd362a"}
Jul 28 14:00:29.309737 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.309460    2558 scope.go:117] "RemoveContainer" containerID="fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"
Jul 28 14:00:29.317734 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.317701    2558 scope.go:117] "RemoveContainer" containerID="a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"
Jul 28 14:00:29.327461 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.327439    2558 scope.go:117] "RemoveContainer" containerID="ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"
Jul 28 14:00:29.331002 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.330977    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 14:00:29.335588 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.335572    2558 scope.go:117] "RemoveContainer" containerID="fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"
Jul 28 14:00:29.335887 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:00:29.335870    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b\": container with ID starting with fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b not found: ID does not exist" containerID="fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"
Jul 28 14:00:29.335948 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.335896    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b"} err="failed to get container status \"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b\": rpc error: code = NotFound desc = could not find container \"fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b\": container with ID starting with fab914b9913893bb09fe52f9801064bd39fed60e64a835180ba2e9b2b53db30b not found: ID does not exist"
Jul 28 14:00:29.335948 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.335913    2558 scope.go:117] "RemoveContainer" containerID="a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"
Jul 28 14:00:29.336154 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:00:29.336135    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190\": container with ID starting with a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190 not found: ID does not exist" containerID="a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"
Jul 28 14:00:29.336199 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.336160    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190"} err="failed to get container status \"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190\": rpc error: code = NotFound desc = could not find container \"a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190\": container with ID starting with a0735df556c1c178263a0a16a3950957f3ec813a437c8ee203791d70a925c190 not found: ID does not exist"
Jul 28 14:00:29.336199 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.336175    2558 scope.go:117] "RemoveContainer" containerID="ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"
Jul 28 14:00:29.336407 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:00:29.336389    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7\": container with ID starting with ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7 not found: ID does not exist" containerID="ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"
Jul 28 14:00:29.336456 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.336415    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7"} err="failed to get container status \"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7\": rpc error: code = NotFound desc = could not find container \"ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7\": container with ID starting with ac276d9cbae341fcdbf988cb808cd9d4cbf62daf029e1369308ae64dc102f2f7 not found: ID does not exist"
Jul 28 14:00:29.339442 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:29.339422    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-szw26"]
Jul 28 14:00:30.023469 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.023430    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" path="/var/lib/kubelet/pods/acc2185e-7bce-4dbf-9abe-901e9d3a0d4b/volumes"
Jul 28 14:00:30.047363 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047326    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:00:30.047619 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047607    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container"
Jul 28 14:00:30.047685 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047620    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container"
Jul 28 14:00:30.047685 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047638    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="storage-initializer"
Jul 28 14:00:30.047685 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047644    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="storage-initializer"
Jul 28 14:00:30.047685 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047658    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kube-rbac-proxy"
Jul 28 14:00:30.047685 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047664    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kube-rbac-proxy"
Jul 28 14:00:30.047972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047704    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kube-rbac-proxy"
Jul 28 14:00:30.047972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.047730    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc2185e-7bce-4dbf-9abe-901e9d3a0d4b" containerName="kserve-container"
Jul 28 14:00:30.052207 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.052187    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.055188 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.055166    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 14:00:30.055188 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.055181    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 14:00:30.055359 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.055241    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 14:00:30.055445 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.055433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 14:00:30.056517 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.056500    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:00:30.056584 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.056528    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:00:30.070348 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.070315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.070467 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.070352    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.070467 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.070383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wbh\" (UniqueName: \"kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.070683 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.070665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.071198 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.070940    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:00:30.171655 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.171615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.171655 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.171649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.171953 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.171672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wbh\" (UniqueName: \"kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.171953 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.171737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.171953 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:00:30.171801    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 28 14:00:30.171953 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:00:30.171877    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls podName:ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14 nodeName:}" failed. No retries permitted until 2026-07-28 14:00:30.671856935 +0000 UTC m=+5137.157481505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls") pod "isvc-sklearn-s3-predictor-6969f6ff49-wspbb" (UID: "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 28 14:00:30.172200 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.172181    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.172378 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.172361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.181567 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.181548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wbh\" (UniqueName: \"kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.675451 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.675417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.677969 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.677946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6969f6ff49-wspbb\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:30.961935 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:30.961850    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:31.097097 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:31.097071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:00:31.098966 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:00:31.098934    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5ca1a6_7c9c_4a0a_992e_52de04cb4c14.slice/crio-2d8294d9dc22c27d745fe796cd19c8cf9754d3a7fce1790267e0449bac0609ba WatchSource:0}: Error finding container 2d8294d9dc22c27d745fe796cd19c8cf9754d3a7fce1790267e0449bac0609ba: Status 404 returned error can't find the container with id 2d8294d9dc22c27d745fe796cd19c8cf9754d3a7fce1790267e0449bac0609ba
Jul 28 14:00:31.320617 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:31.320518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerStarted","Data":"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"}
Jul 28 14:00:31.320617 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:31.320573    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerStarted","Data":"2d8294d9dc22c27d745fe796cd19c8cf9754d3a7fce1790267e0449bac0609ba"}
Jul 28 14:00:32.326342 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:32.326254    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerID="f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf" exitCode=0
Jul 28 14:00:32.326691 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:32.326350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerDied","Data":"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"}
Jul 28 14:00:33.332119 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:33.332084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerStarted","Data":"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"}
Jul 28 14:00:33.332492 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:33.332146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerStarted","Data":"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"}
Jul 28 14:00:33.332492 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:33.332233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:33.354140 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:33.354092    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podStartSLOduration=2.448211743 podStartE2EDuration="3.354076373s" totalImagesPullingTime="0s" totalInitContainerRuntime="905.86463ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:00:33.351663288 +0000 UTC m=+5139.837287879" watchObservedRunningTime="2026-07-28 14:00:33.354076373 +0000 UTC m=+5139.839700986"
Jul 28 14:00:34.336258 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:34.336225    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:34.337527 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:34.337497    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:00:35.340025 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:35.339986    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:00:40.344938 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:40.344896    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:00:40.345515 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:40.345485    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:00:50.346016 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:00:50.345974    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:00.345933 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:00.345888    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:10.345402 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:10.345359    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:20.346360 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:20.346317    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:30.346085 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:30.346046    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:40.345479 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:40.345441    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:01:50.346973 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:01:50.346896    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:02:00.221704 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.221669    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:02:00.222220 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.221942    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" containerID="cri-o://3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971" gracePeriod=30
Jul 28 14:02:00.222220 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.221969    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kube-rbac-proxy" containerID="cri-o://a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8" gracePeriod=30
Jul 28 14:02:00.340552 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.340504    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 28 14:02:00.346011 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.345983    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 14:02:00.389543 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.389513    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:02:00.392954 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.392938    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.395732 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.395693    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 14:02:00.395869 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.395816    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:02:00.396010 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.395994    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:02:00.405442 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.405421    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:02:00.475520 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.475414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.475520 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.475460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.475520 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.475488    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.475520 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.475507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtstj\" (UniqueName: \"kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.475850 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.475626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.576945 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.576913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577155 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.576955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577155 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.576973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577155 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.576992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtstj\" (UniqueName: \"kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577155 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.577046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577451 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.577427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577744 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.577665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.577744 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.577690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.579504 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.579485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.586750 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.586707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtstj\" (UniqueName: \"kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.702526 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.702491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:00.703949 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.703921    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerID="a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8" exitCode=2
Jul 28 14:02:00.704065 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.703997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerDied","Data":"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"}
Jul 28 14:02:00.838168 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:00.838137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:02:00.840626 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:02:00.840600    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d6c254_a43c_463a_a1da_90fd86c88d3f.slice/crio-4e7daef38310659691292ddc1688652de7caf8ba337d3678f1d768624f61228e WatchSource:0}: Error finding container 4e7daef38310659691292ddc1688652de7caf8ba337d3678f1d768624f61228e: Status 404 returned error can't find the container with id 4e7daef38310659691292ddc1688652de7caf8ba337d3678f1d768624f61228e
Jul 28 14:02:01.709700 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:01.709662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerStarted","Data":"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"}
Jul 28 14:02:01.710191 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:01.709748    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerStarted","Data":"4e7daef38310659691292ddc1688652de7caf8ba337d3678f1d768624f61228e"}
Jul 28 14:02:02.720368 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:02.720326    2558 generic.go:356] "Generic (PLEG): container finished" podID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerID="4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937" exitCode=0
Jul 28 14:02:02.720854 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:02.720405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerDied","Data":"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"}
Jul 28 14:02:03.726322 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:03.726288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerStarted","Data":"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"}
Jul 28 14:02:03.726708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:03.726343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerStarted","Data":"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"}
Jul 28 14:02:03.726708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:03.726412    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:03.748150 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:03.748100    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podStartSLOduration=2.8085829 podStartE2EDuration="3.748087359s" totalImagesPullingTime="0s" totalInitContainerRuntime="939.504459ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:02:03.746027357 +0000 UTC m=+5230.231651954" watchObservedRunningTime="2026-07-28 14:02:03.748087359 +0000 UTC m=+5230.233711951"
Jul 28 14:02:04.730793 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:04.730759    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:04.732021 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:04.731991    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:05.169439 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.169412    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:02:05.218463 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218431    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location\") pod \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") "
Jul 28 14:02:05.218634 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218492    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config\") pod \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") "
Jul 28 14:02:05.218634 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79wbh\" (UniqueName: \"kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh\") pod \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") "
Jul 28 14:02:05.218634 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218562    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") pod \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\" (UID: \"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14\") "
Jul 28 14:02:05.218834 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218758    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" (UID: "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:02:05.218834 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.218800    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" (UID: "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:02:05.221043 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.221016    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh" (OuterVolumeSpecName: "kube-api-access-79wbh") pod "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" (UID: "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14"). InnerVolumeSpecName "kube-api-access-79wbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:02:05.221158 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.221083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" (UID: "ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:02:05.319928 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.319844    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wbh\" (UniqueName: \"kubernetes.io/projected/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-api-access-79wbh\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:02:05.319928 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.319877    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:02:05.319928 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.319887    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:02:05.319928 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.319895    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:02:05.736732 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.736680    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerID="3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971" exitCode=0
Jul 28 14:02:05.737132 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.736738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerDied","Data":"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"}
Jul 28 14:02:05.737132 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.736758    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"
Jul 28 14:02:05.737132 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.736798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb" event={"ID":"ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14","Type":"ContainerDied","Data":"2d8294d9dc22c27d745fe796cd19c8cf9754d3a7fce1790267e0449bac0609ba"}
Jul 28 14:02:05.737132 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.736826    2558 scope.go:117] "RemoveContainer" containerID="a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"
Jul 28 14:02:05.737363 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.737336    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:05.745354 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.745337    2558 scope.go:117] "RemoveContainer" containerID="3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"
Jul 28 14:02:05.752517 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.752498    2558 scope.go:117] "RemoveContainer" containerID="f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"
Jul 28 14:02:05.758787 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.758765    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:02:05.760401 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.760383    2558 scope.go:117] "RemoveContainer" containerID="a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"
Jul 28 14:02:05.761065 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:02:05.761040    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8\": container with ID starting with a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8 not found: ID does not exist" containerID="a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"
Jul 28 14:02:05.761162 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.761074    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8"} err="failed to get container status \"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8\": rpc error: code = NotFound desc = could not find container \"a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8\": container with ID starting with a5c8145e7a2e0e1bfcc3d48f89d074099724a9455b543c993a893d9854e475b8 not found: ID does not exist"
Jul 28 14:02:05.761162 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.761095    2558 scope.go:117] "RemoveContainer" containerID="3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"
Jul 28 14:02:05.761386 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:02:05.761365    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971\": container with ID starting with 3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971 not found: ID does not exist" containerID="3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"
Jul 28 14:02:05.761425 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.761395    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971"} err="failed to get container status \"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971\": rpc error: code = NotFound desc = could not find container \"3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971\": container with ID starting with 3c54099aab96aa721214108ca9ce39bee1a15950b269bd45084aa2d1c2e79971 not found: ID does not exist"
Jul 28 14:02:05.761425 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.761411    2558 scope.go:117] "RemoveContainer" containerID="f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"
Jul 28 14:02:05.761655 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:02:05.761633    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf\": container with ID starting with f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf not found: ID does not exist" containerID="f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"
Jul 28 14:02:05.761707 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.761664    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf"} err="failed to get container status \"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf\": rpc error: code = NotFound desc = could not find container \"f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf\": container with ID starting with f918f3453729da9b93d745e679e148b23d06782496f9b141a63e2d8449315dcf not found: ID does not exist"
Jul 28 14:02:05.762274 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:05.762257    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6969f6ff49-wspbb"]
Jul 28 14:02:06.017897 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:06.017814    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" path="/var/lib/kubelet/pods/ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14/volumes"
Jul 28 14:02:10.741780 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:10.741750    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:02:10.742242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:10.742210    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:20.742897 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:20.742856    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:30.742323 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:30.742276    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:40.742814 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:40.742769    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:02:50.742737 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:02:50.742679    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:03:00.742764 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:00.742701    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:03:10.742471 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:10.742433    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:03:20.742902 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:20.742873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:03:30.414889 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:30.414849    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:03:30.415301 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:30.415154    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" containerID="cri-o://188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0" gracePeriod=30
Jul 28 14:03:30.415301 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:30.415171    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kube-rbac-proxy" containerID="cri-o://12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893" gracePeriod=30
Jul 28 14:03:30.738569 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:30.738473    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 14:03:30.742791 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:30.742757    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 14:03:31.078429 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.078346    2558 generic.go:356] "Generic (PLEG): container finished" podID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerID="12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893" exitCode=2
Jul 28 14:03:31.078571 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.078423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerDied","Data":"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"}
Jul 28 14:03:31.682317 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682277    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682618    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kube-rbac-proxy"
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682633    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kube-rbac-proxy"
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682653    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="storage-initializer"
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682659    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="storage-initializer"
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682665    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container"
Jul 28 14:03:31.682692 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682671    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container"
Jul 28 14:03:31.682906 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682742    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kserve-container"
Jul 28 14:03:31.682906 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.682750    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ca1a6-7c9c-4a0a-992e-52de04cb4c14" containerName="kube-rbac-proxy"
Jul 28 14:03:31.685954 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.685933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.688820 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.688797    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 14:03:31.688965 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.688801    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:03:31.699663 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.699639    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:31.835981 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.835951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.836177 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.836013    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzt4\" (UniqueName: \"kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.836177 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.836109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.836177 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.836166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.937553 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.937454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzt4\" (UniqueName: \"kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.937553 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.937507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.937844 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.937643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.937844 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.937743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.937926 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.937872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.938310 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.938288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.940176 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.940159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.947065 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.947043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzt4\" (UniqueName: \"kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:31.996745 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:31.996678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:32.140486 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:32.136539    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:32.142081 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:03:32.142053    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d2a1e9_ae2c_43bb_add0_3c7983e4eaa3.slice/crio-753cde99e87692f0e628464e10e0591055013872a4026a147f5f793e3f6d6c9e WatchSource:0}: Error finding container 753cde99e87692f0e628464e10e0591055013872a4026a147f5f793e3f6d6c9e: Status 404 returned error can't find the container with id 753cde99e87692f0e628464e10e0591055013872a4026a147f5f793e3f6d6c9e
Jul 28 14:03:33.090036 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:33.089990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerStarted","Data":"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"}
Jul 28 14:03:33.090479 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:33.090060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerStarted","Data":"753cde99e87692f0e628464e10e0591055013872a4026a147f5f793e3f6d6c9e"}
Jul 28 14:03:35.261837 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.261807    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:03:35.262188 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262041    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config\") pod \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") "
Jul 28 14:03:35.262188 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtstj\" (UniqueName: \"kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj\") pod \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") "
Jul 28 14:03:35.262300 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262202    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls\") pod \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") "
Jul 28 14:03:35.262300 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert\") pod \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") "
Jul 28 14:03:35.262300 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262291    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location\") pod \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\" (UID: \"c7d6c254-a43c-463a-a1da-90fd86c88d3f\") "
Jul 28 14:03:35.262461 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262421    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7d6c254-a43c-463a-a1da-90fd86c88d3f" (UID: "c7d6c254-a43c-463a-a1da-90fd86c88d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:03:35.262553 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262533    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:35.262660 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7d6c254-a43c-463a-a1da-90fd86c88d3f" (UID: "c7d6c254-a43c-463a-a1da-90fd86c88d3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:03:35.262660 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.262619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7d6c254-a43c-463a-a1da-90fd86c88d3f" (UID: "c7d6c254-a43c-463a-a1da-90fd86c88d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:03:35.264343 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.264323    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj" (OuterVolumeSpecName: "kube-api-access-vtstj") pod "c7d6c254-a43c-463a-a1da-90fd86c88d3f" (UID: "c7d6c254-a43c-463a-a1da-90fd86c88d3f"). InnerVolumeSpecName "kube-api-access-vtstj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:03:35.264343 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.264317    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7d6c254-a43c-463a-a1da-90fd86c88d3f" (UID: "c7d6c254-a43c-463a-a1da-90fd86c88d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:03:35.363582 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.363555    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtstj\" (UniqueName: \"kubernetes.io/projected/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kube-api-access-vtstj\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:35.363582 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.363583    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7d6c254-a43c-463a-a1da-90fd86c88d3f-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:35.363806 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.363593    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d6c254-a43c-463a-a1da-90fd86c88d3f-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:35.363806 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:35.363601    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d6c254-a43c-463a-a1da-90fd86c88d3f-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:36.104466 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.104432    2558 generic.go:356] "Generic (PLEG): container finished" podID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerID="188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0" exitCode=0
Jul 28 14:03:36.104624 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.104489    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"
Jul 28 14:03:36.104624 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.104496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerDied","Data":"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"}
Jul 28 14:03:36.104624 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.104552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7" event={"ID":"c7d6c254-a43c-463a-a1da-90fd86c88d3f","Type":"ContainerDied","Data":"4e7daef38310659691292ddc1688652de7caf8ba337d3678f1d768624f61228e"}
Jul 28 14:03:36.104624 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.104587    2558 scope.go:117] "RemoveContainer" containerID="12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"
Jul 28 14:03:36.113683 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.113662    2558 scope.go:117] "RemoveContainer" containerID="188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"
Jul 28 14:03:36.121588 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.121565    2558 scope.go:117] "RemoveContainer" containerID="4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"
Jul 28 14:03:36.123064 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.123036    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:03:36.128453 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.128429    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-545f455d97-b4nd7"]
Jul 28 14:03:36.130240 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130226    2558 scope.go:117] "RemoveContainer" containerID="12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"
Jul 28 14:03:36.130487 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:36.130468    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893\": container with ID starting with 12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893 not found: ID does not exist" containerID="12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"
Jul 28 14:03:36.130551 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130496    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893"} err="failed to get container status \"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893\": rpc error: code = NotFound desc = could not find container \"12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893\": container with ID starting with 12229e0d27139ed5bebb93c4763a00db98d09b92b735af54010156cfcd7cc893 not found: ID does not exist"
Jul 28 14:03:36.130551 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130514    2558 scope.go:117] "RemoveContainer" containerID="188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"
Jul 28 14:03:36.130750 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:36.130714    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0\": container with ID starting with 188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0 not found: ID does not exist" containerID="188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"
Jul 28 14:03:36.130854 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130753    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0"} err="failed to get container status \"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0\": rpc error: code = NotFound desc = could not find container \"188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0\": container with ID starting with 188486e7905a0fd5c10347c0dedf043658f23e23f20d08e577b9a9c04ad2a0a0 not found: ID does not exist"
Jul 28 14:03:36.130854 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130766    2558 scope.go:117] "RemoveContainer" containerID="4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"
Jul 28 14:03:36.130977 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:36.130963    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937\": container with ID starting with 4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937 not found: ID does not exist" containerID="4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"
Jul 28 14:03:36.131016 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:36.130979    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937"} err="failed to get container status \"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937\": rpc error: code = NotFound desc = could not find container \"4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937\": container with ID starting with 4adadc1aedb66d13f926d7c6da8c21fdad3b5890a7a9cee30f8134e5589c9937 not found: ID does not exist"
Jul 28 14:03:38.017180 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:38.017143    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" path="/var/lib/kubelet/pods/c7d6c254-a43c-463a-a1da-90fd86c88d3f/volumes"
Jul 28 14:03:38.115453 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:38.115421    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/0.log"
Jul 28 14:03:38.115611 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:38.115464    2558 generic.go:356] "Generic (PLEG): container finished" podID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerID="05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a" exitCode=1
Jul 28 14:03:38.115611 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:38.115543    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerDied","Data":"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"}
Jul 28 14:03:39.120790 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:39.120767    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/0.log"
Jul 28 14:03:39.121169 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:39.120859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerStarted","Data":"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"}
Jul 28 14:03:41.541533 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:41.541497    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:41.541960 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:41.541705    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer" containerID="cri-o://e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4" gracePeriod=30
Jul 28 14:03:42.672786 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.672746    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673092    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673104    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673112    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="storage-initializer"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673117    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="storage-initializer"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673128    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kube-rbac-proxy"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673134    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kube-rbac-proxy"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673200    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kube-rbac-proxy"
Jul 28 14:03:42.673242 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.673213    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d6c254-a43c-463a-a1da-90fd86c88d3f" containerName="kserve-container"
Jul 28 14:03:42.676513 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.676493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.681121 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.681104    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 14:03:42.681267 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.681242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:03:42.681267 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.681262    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:03:42.691670 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.691644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:03:42.707122 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.707103    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/1.log"
Jul 28 14:03:42.707525 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.707509    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/0.log"
Jul 28 14:03:42.709820 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.709802    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:42.722492 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722471    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location\") pod \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") "
Jul 28 14:03:42.722591 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls\") pod \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") "
Jul 28 14:03:42.722591 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config\") pod \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") "
Jul 28 14:03:42.722712 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tzt4\" (UniqueName: \"kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4\") pod \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\" (UID: \"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3\") "
Jul 28 14:03:42.722712 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.722844 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.722844 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722771    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" (UID: "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:03:42.722844 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.723001 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.723001 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lbl\" (UniqueName: \"kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.723001 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722942    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" (UID: "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:03:42.723001 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.722981    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:42.724793 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.724771    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" (UID: "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:03:42.724907 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.724894    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4" (OuterVolumeSpecName: "kube-api-access-8tzt4") pod "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" (UID: "34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3"). InnerVolumeSpecName "kube-api-access-8tzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:03:42.824341 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.824534 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.824534 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.824534 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.824534 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lbl\" (UniqueName: \"kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.824534 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824531    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:42.824879 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824546    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:42.824879 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:42.824559    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 14:03:42.824879 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:42.824636    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls podName:d74813c0-216b-4f9f-87d3-71d6592d8170 nodeName:}" failed. No retries permitted until 2026-07-28 14:03:43.324614403 +0000 UTC m=+5329.810238977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 14:03:42.824879 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824561    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzt4\" (UniqueName: \"kubernetes.io/projected/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3-kube-api-access-8tzt4\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:03:42.824879 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.824827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.825107 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.825093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.825240 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.825218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:42.833165 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:42.833140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lbl\" (UniqueName: \"kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:43.138941 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.138912    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/1.log"
Jul 28 14:03:43.139307 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139289    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4_34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/storage-initializer/0.log"
Jul 28 14:03:43.139433 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139328    2558 generic.go:356] "Generic (PLEG): container finished" podID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerID="e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4" exitCode=1
Jul 28 14:03:43.139433 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139377    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"
Jul 28 14:03:43.139433 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139409    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerDied","Data":"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"}
Jul 28 14:03:43.139549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139482    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4" event={"ID":"34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3","Type":"ContainerDied","Data":"753cde99e87692f0e628464e10e0591055013872a4026a147f5f793e3f6d6c9e"}
Jul 28 14:03:43.139549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.139509    2558 scope.go:117] "RemoveContainer" containerID="e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"
Jul 28 14:03:43.147860 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.147842    2558 scope.go:117] "RemoveContainer" containerID="05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"
Jul 28 14:03:43.154965 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.154951    2558 scope.go:117] "RemoveContainer" containerID="e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"
Jul 28 14:03:43.155190 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:43.155173    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4\": container with ID starting with e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4 not found: ID does not exist" containerID="e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"
Jul 28 14:03:43.155246 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.155197    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4"} err="failed to get container status \"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4\": rpc error: code = NotFound desc = could not find container \"e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4\": container with ID starting with e9c4762a9896368b94292b16e6652dedf5fc4f2f3d7078ba1b7281dec58fccf4 not found: ID does not exist"
Jul 28 14:03:43.155246 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.155215    2558 scope.go:117] "RemoveContainer" containerID="05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"
Jul 28 14:03:43.155414 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:03:43.155398    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a\": container with ID starting with 05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a not found: ID does not exist" containerID="05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"
Jul 28 14:03:43.155457 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.155418    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a"} err="failed to get container status \"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a\": rpc error: code = NotFound desc = could not find container \"05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a\": container with ID starting with 05e50b8cd290fc46384edc0380081835ec372437f5d83a7b9c09197ca119272a not found: ID does not exist"
Jul 28 14:03:43.181976 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.181947    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:43.196000 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.195967    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6fd8968769-7c4q4"]
Jul 28 14:03:43.328285 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.328250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:43.330768 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.330746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:43.605282 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.605245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:43.738598 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:43.738573    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:03:43.741122 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:03:43.741096    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74813c0_216b_4f9f_87d3_71d6592d8170.slice/crio-8f84fac9dbf3bef09469bb6c67c1f719469e1359534cf532f9647364f74f8854 WatchSource:0}: Error finding container 8f84fac9dbf3bef09469bb6c67c1f719469e1359534cf532f9647364f74f8854: Status 404 returned error can't find the container with id 8f84fac9dbf3bef09469bb6c67c1f719469e1359534cf532f9647364f74f8854
Jul 28 14:03:44.018304 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:44.018210    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" path="/var/lib/kubelet/pods/34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3/volumes"
Jul 28 14:03:44.146519 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:44.146479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerStarted","Data":"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"}
Jul 28 14:03:44.146663 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:44.146533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerStarted","Data":"8f84fac9dbf3bef09469bb6c67c1f719469e1359534cf532f9647364f74f8854"}
Jul 28 14:03:45.151973 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:45.151944    2558 generic.go:356] "Generic (PLEG): container finished" podID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerID="b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab" exitCode=0
Jul 28 14:03:45.152374 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:45.152038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerDied","Data":"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"}
Jul 28 14:03:46.158385 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.158354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerStarted","Data":"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"}
Jul 28 14:03:46.158760 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.158412    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:46.158760 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.158432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerStarted","Data":"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"}
Jul 28 14:03:46.158760 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.158554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:46.160020 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.159995    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:03:46.181746 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:46.181680    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podStartSLOduration=3.25352123 podStartE2EDuration="4.18166558s" totalImagesPullingTime="0s" totalInitContainerRuntime="928.14435ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:03:46.18056997 +0000 UTC m=+5332.666194562" watchObservedRunningTime="2026-07-28 14:03:46.18166558 +0000 UTC m=+5332.667290171"
Jul 28 14:03:47.162387 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:47.162346    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:03:52.167375 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:52.167341    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:03:52.167990 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:03:52.167879    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:02.168196 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:02.168153    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:12.168130 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:12.168087    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:22.168504 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:22.168459    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:32.168139 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:32.168097    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:42.168155 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:42.168116    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:52.168379 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:52.168336    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 14:04:55.294446 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:55.294347    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 14:04:55.322289 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:04:55.322263    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 14:05:00.017666 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:00.017636    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:05:02.718754 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:02.718649    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:05:02.719185 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:02.718975    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container" containerID="cri-o://6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca" gracePeriod=30
Jul 28 14:05:02.719185 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:02.719015    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kube-rbac-proxy" containerID="cri-o://c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3" gracePeriod=30
Jul 28 14:05:03.488749 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.488699    2558 generic.go:356] "Generic (PLEG): container finished" podID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerID="c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3" exitCode=2
Jul 28 14:05:03.488926 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.488752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerDied","Data":"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"}
Jul 28 14:05:03.850753 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.850706    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851025    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851035    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851052    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851058    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851114    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.851129 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.851121    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d2a1e9-ae2c-43bb-add0-3c7983e4eaa3" containerName="storage-initializer"
Jul 28 14:05:03.854146 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.854126    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:03.859002 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.858979    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:05:03.859838 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.859820    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 14:05:03.873165 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:03.873139    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:04.005135 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.005097    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.005326 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.005165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.005326 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.005210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zlf\" (UniqueName: \"kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.005326 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.005281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.106594 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.106502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.106594 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.106535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zlf\" (UniqueName: \"kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.106594 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.106564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.106944 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.106644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.107032 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.107008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.107392 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.107369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.109214 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.109188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.115760 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.115711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zlf\" (UniqueName: \"kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.165074 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.165029    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:04.305297 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.305271    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:04.308182 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:05:04.308156    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62fda11_dc79_4776_b8c2_cb7a318ac6a0.slice/crio-0262004bd42cf320b5560e3550c8ea719767c65ec083d062903193c7942cbbe2 WatchSource:0}: Error finding container 0262004bd42cf320b5560e3550c8ea719767c65ec083d062903193c7942cbbe2: Status 404 returned error can't find the container with id 0262004bd42cf320b5560e3550c8ea719767c65ec083d062903193c7942cbbe2
Jul 28 14:05:04.494556 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.494509    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerStarted","Data":"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"}
Jul 28 14:05:04.494756 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:04.494594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerStarted","Data":"0262004bd42cf320b5560e3550c8ea719767c65ec083d062903193c7942cbbe2"}
Jul 28 14:05:07.163266 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.163212    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 28 14:05:07.869093 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.869067    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:05:07.941068 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.940969    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") pod \"d74813c0-216b-4f9f-87d3-71d6592d8170\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") "
Jul 28 14:05:07.941068 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941008    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4lbl\" (UniqueName: \"kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl\") pod \"d74813c0-216b-4f9f-87d3-71d6592d8170\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") "
Jul 28 14:05:07.941068 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941034    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert\") pod \"d74813c0-216b-4f9f-87d3-71d6592d8170\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") "
Jul 28 14:05:07.941373 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config\") pod \"d74813c0-216b-4f9f-87d3-71d6592d8170\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") "
Jul 28 14:05:07.941373 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location\") pod \"d74813c0-216b-4f9f-87d3-71d6592d8170\" (UID: \"d74813c0-216b-4f9f-87d3-71d6592d8170\") "
Jul 28 14:05:07.941472 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941366    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d74813c0-216b-4f9f-87d3-71d6592d8170" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:05:07.941641 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941613    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d74813c0-216b-4f9f-87d3-71d6592d8170" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:05:07.941738 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941622    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:07.941738 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.941650    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d74813c0-216b-4f9f-87d3-71d6592d8170" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:05:07.943335 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.943314    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d74813c0-216b-4f9f-87d3-71d6592d8170" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:05:07.943441 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:07.943344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl" (OuterVolumeSpecName: "kube-api-access-f4lbl") pod "d74813c0-216b-4f9f-87d3-71d6592d8170" (UID: "d74813c0-216b-4f9f-87d3-71d6592d8170"). InnerVolumeSpecName "kube-api-access-f4lbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:05:08.042404 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.042367    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:08.042404 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.042396    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74813c0-216b-4f9f-87d3-71d6592d8170-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:08.042404 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.042407    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74813c0-216b-4f9f-87d3-71d6592d8170-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:08.042627 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.042418    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lbl\" (UniqueName: \"kubernetes.io/projected/d74813c0-216b-4f9f-87d3-71d6592d8170-kube-api-access-f4lbl\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:08.513688 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.513652    2558 generic.go:356] "Generic (PLEG): container finished" podID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerID="6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca" exitCode=0
Jul 28 14:05:08.514234 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.513699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerDied","Data":"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"}
Jul 28 14:05:08.514234 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.513710    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"
Jul 28 14:05:08.514234 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.513781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9" event={"ID":"d74813c0-216b-4f9f-87d3-71d6592d8170","Type":"ContainerDied","Data":"8f84fac9dbf3bef09469bb6c67c1f719469e1359534cf532f9647364f74f8854"}
Jul 28 14:05:08.514234 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.513821    2558 scope.go:117] "RemoveContainer" containerID="c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"
Jul 28 14:05:08.522197 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.522170    2558 scope.go:117] "RemoveContainer" containerID="6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"
Jul 28 14:05:08.529423 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.529403    2558 scope.go:117] "RemoveContainer" containerID="b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"
Jul 28 14:05:08.533941 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.533920    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:05:08.537262 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.537245    2558 scope.go:117] "RemoveContainer" containerID="c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"
Jul 28 14:05:08.537504 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:08.537486    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3\": container with ID starting with c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3 not found: ID does not exist" containerID="c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"
Jul 28 14:05:08.537572 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.537513    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3"} err="failed to get container status \"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3\": rpc error: code = NotFound desc = could not find container \"c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3\": container with ID starting with c0f202e0ff870c1d5b192a08ba749ad63798d9cac03f8ad47c3591215fb20fa3 not found: ID does not exist"
Jul 28 14:05:08.537572 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.537530    2558 scope.go:117] "RemoveContainer" containerID="6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"
Jul 28 14:05:08.537802 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:08.537787    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca\": container with ID starting with 6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca not found: ID does not exist" containerID="6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"
Jul 28 14:05:08.537859 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.537807    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca"} err="failed to get container status \"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca\": rpc error: code = NotFound desc = could not find container \"6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca\": container with ID starting with 6248e3a761dead95c75c5a2626442e36d4a3a6329af544356515ee4cd28998ca not found: ID does not exist"
Jul 28 14:05:08.537859 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.537819    2558 scope.go:117] "RemoveContainer" containerID="b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"
Jul 28 14:05:08.538057 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:08.538043    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab\": container with ID starting with b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab not found: ID does not exist" containerID="b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"
Jul 28 14:05:08.538093 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.538059    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab"} err="failed to get container status \"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab\": rpc error: code = NotFound desc = could not find container \"b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab\": container with ID starting with b03590ec47855f18a981e9c7016b08614eba5fb5deb63aec95043b7efb4ec5ab not found: ID does not exist"
Jul 28 14:05:08.538170 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:08.538154    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-8b6fb7d89-x2pl9"]
Jul 28 14:05:10.017664 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:10.017626    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" path="/var/lib/kubelet/pods/d74813c0-216b-4f9f-87d3-71d6592d8170/volumes"
Jul 28 14:05:10.525252 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:10.525221    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/0.log"
Jul 28 14:05:10.525448 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:10.525264    2558 generic.go:356] "Generic (PLEG): container finished" podID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerID="c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956" exitCode=1
Jul 28 14:05:10.525448 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:10.525319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerDied","Data":"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"}
Jul 28 14:05:11.531348 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:11.531267    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/0.log"
Jul 28 14:05:11.531348 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:11.531329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerStarted","Data":"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"}
Jul 28 14:05:13.796657 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:13.796619    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:13.797237 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:13.796949    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer" containerID="cri-o://52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3" gracePeriod=30
Jul 28 14:05:14.877240 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877196    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877554    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877564    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877571    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kube-rbac-proxy"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877576    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kube-rbac-proxy"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877586    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="storage-initializer"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877592    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="storage-initializer"
Jul 28 14:05:14.877637 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877640    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kube-rbac-proxy"
Jul 28 14:05:14.877890 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.877651    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74813c0-216b-4f9f-87d3-71d6592d8170" containerName="kserve-container"
Jul 28 14:05:14.882148 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.882125    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:14.885191 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.885165    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:05:14.885691 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.885673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 14:05:14.885760 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.885673    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:05:14.893165 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:14.893138    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:05:15.004879 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.004837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.005061 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.004977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.005061 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.005017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.005061 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.005045    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwk9\" (UniqueName: \"kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.005196 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.005070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105513 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwk9\" (UniqueName: \"kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105708 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.105993 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:15.105803    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 14:05:15.105993 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:15.105884    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls podName:5cf31373-9f7f-4ea3-8523-d58b54d6d3d1 nodeName:}" failed. No retries permitted until 2026-07-28 14:05:15.605863317 +0000 UTC m=+5422.091487889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 14:05:15.105993 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.105936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.106225 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.106205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.106368 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.106347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.114287 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.114261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwk9\" (UniqueName: \"kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.611493 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.611447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.614199 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.614160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.795517 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.795469    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:15.936556 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:15.936528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:05:15.938829 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:05:15.938792    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf31373_9f7f_4ea3_8523_d58b54d6d3d1.slice/crio-199bb036d129df6b851417aebd620f86b1a174aff65777dd959b61d2fd7f3be2 WatchSource:0}: Error finding container 199bb036d129df6b851417aebd620f86b1a174aff65777dd959b61d2fd7f3be2: Status 404 returned error can't find the container with id 199bb036d129df6b851417aebd620f86b1a174aff65777dd959b61d2fd7f3be2
Jul 28 14:05:16.341816 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.341786    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/1.log"
Jul 28 14:05:16.342207 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.342189    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/0.log"
Jul 28 14:05:16.344654 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.344634    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:16.520678 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.520564    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58zlf\" (UniqueName: \"kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf\") pod \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") "
Jul 28 14:05:16.520678 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.520621    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls\") pod \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") "
Jul 28 14:05:16.520970 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.520683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config\") pod \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") "
Jul 28 14:05:16.520970 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.520710    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location\") pod \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\" (UID: \"a62fda11-dc79-4776-b8c2-cb7a318ac6a0\") "
Jul 28 14:05:16.521076 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.521044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a62fda11-dc79-4776-b8c2-cb7a318ac6a0" (UID: "a62fda11-dc79-4776-b8c2-cb7a318ac6a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:05:16.521127 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.521112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a62fda11-dc79-4776-b8c2-cb7a318ac6a0" (UID: "a62fda11-dc79-4776-b8c2-cb7a318ac6a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:05:16.523219 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.523171    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf" (OuterVolumeSpecName: "kube-api-access-58zlf") pod "a62fda11-dc79-4776-b8c2-cb7a318ac6a0" (UID: "a62fda11-dc79-4776-b8c2-cb7a318ac6a0"). InnerVolumeSpecName "kube-api-access-58zlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:05:16.523219 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.523205    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a62fda11-dc79-4776-b8c2-cb7a318ac6a0" (UID: "a62fda11-dc79-4776-b8c2-cb7a318ac6a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:05:16.555502 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.555471    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/1.log"
Jul 28 14:05:16.555921 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.555901    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm_a62fda11-dc79-4776-b8c2-cb7a318ac6a0/storage-initializer/0.log"
Jul 28 14:05:16.556039 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.555946    2558 generic.go:356] "Generic (PLEG): container finished" podID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerID="52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3" exitCode=1
Jul 28 14:05:16.556039 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.556007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"
Jul 28 14:05:16.556156 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.556051    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerDied","Data":"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"}
Jul 28 14:05:16.556156 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.556139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm" event={"ID":"a62fda11-dc79-4776-b8c2-cb7a318ac6a0","Type":"ContainerDied","Data":"0262004bd42cf320b5560e3550c8ea719767c65ec083d062903193c7942cbbe2"}
Jul 28 14:05:16.556263 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.556181    2558 scope.go:117] "RemoveContainer" containerID="52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"
Jul 28 14:05:16.557942 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.557912    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerStarted","Data":"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"}
Jul 28 14:05:16.558076 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.557982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerStarted","Data":"199bb036d129df6b851417aebd620f86b1a174aff65777dd959b61d2fd7f3be2"}
Jul 28 14:05:16.566027 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.566004    2558 scope.go:117] "RemoveContainer" containerID="c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"
Jul 28 14:05:16.576179 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.576155    2558 scope.go:117] "RemoveContainer" containerID="52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"
Jul 28 14:05:16.576513 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:16.576493    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3\": container with ID starting with 52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3 not found: ID does not exist" containerID="52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"
Jul 28 14:05:16.576578 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.576524    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3"} err="failed to get container status \"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3\": rpc error: code = NotFound desc = could not find container \"52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3\": container with ID starting with 52f64d15fbd14fd2422306fd11f051578dd1bf9711e52d40ec9bd7388217c0c3 not found: ID does not exist"
Jul 28 14:05:16.576578 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.576548    2558 scope.go:117] "RemoveContainer" containerID="c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"
Jul 28 14:05:16.576899 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:05:16.576876    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956\": container with ID starting with c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956 not found: ID does not exist" containerID="c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"
Jul 28 14:05:16.576986 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.576906    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956"} err="failed to get container status \"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956\": rpc error: code = NotFound desc = could not find container \"c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956\": container with ID starting with c4e309aa9f74e98c5b85e3058f28152d92568c3b72fc458fe5a3f32e6d826956 not found: ID does not exist"
Jul 28 14:05:16.611646 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.611603    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:16.616986 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.616946    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5bcc9d965-rlmzm"]
Jul 28 14:05:16.621648 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.621612    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:16.621648 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.621648    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:16.621842 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.621664    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zlf\" (UniqueName: \"kubernetes.io/projected/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-kube-api-access-58zlf\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:16.621842 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:16.621678    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a62fda11-dc79-4776-b8c2-cb7a318ac6a0-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:05:17.565189 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:17.565140    2558 generic.go:356] "Generic (PLEG): container finished" podID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerID="1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038" exitCode=0
Jul 28 14:05:17.565623 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:17.565212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerDied","Data":"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"}
Jul 28 14:05:18.018191 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.018151    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" path="/var/lib/kubelet/pods/a62fda11-dc79-4776-b8c2-cb7a318ac6a0/volumes"
Jul 28 14:05:18.571766 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.571730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerStarted","Data":"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"}
Jul 28 14:05:18.572167 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.571802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerStarted","Data":"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"}
Jul 28 14:05:18.572167 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.571829    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:18.572167 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.571850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:18.573540 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.573507    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:05:18.592182 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:18.592103    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podStartSLOduration=3.547673256 podStartE2EDuration="4.592081681s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.044408425s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:05:18.59124812 +0000 UTC m=+5425.076872713" watchObservedRunningTime="2026-07-28 14:05:18.592081681 +0000 UTC m=+5425.077706275"
Jul 28 14:05:19.576124 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:19.576084    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:05:24.580678 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:24.580643    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:05:24.581345 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:24.581312    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:05:34.582302 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:34.582249    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:05:44.581273 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:44.581228    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:05:54.581937 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:05:54.581896    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:04.581878 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:04.581832    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:14.581240 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:14.581199    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:24.581921 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:24.581872    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:25.012580 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:25.012534    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:35.013909 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:35.013870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:06:44.983974 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:44.983933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:06:44.984443 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:44.984216    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" containerID="cri-o://9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653" gracePeriod=30
Jul 28 14:06:44.984443 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:44.984247    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kube-rbac-proxy" containerID="cri-o://343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870" gracePeriod=30
Jul 28 14:06:45.012431 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:45.012395    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 14:06:45.949213 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:45.949180    2558 generic.go:356] "Generic (PLEG): container finished" podID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerID="343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870" exitCode=2
Jul 28 14:06:45.949381 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:45.949253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerDied","Data":"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"}
Jul 28 14:06:46.118080 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118036    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118374    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118386    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118394    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118400    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118451    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.118549 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.118547    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62fda11-dc79-4776-b8c2-cb7a318ac6a0" containerName="storage-initializer"
Jul 28 14:06:46.121679 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.121654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.124446 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.124423    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 14:06:46.124570 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.124516    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:06:46.134781 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.134740    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:06:46.258453 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.258362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.258453 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.258400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.258453 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.258419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.258694 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.258509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kfg\" (UniqueName: \"kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.359860 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.359816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kfg\" (UniqueName: \"kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.360030 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.359895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.360030 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.359921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.360030 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.359937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.360393 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.360368    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.360639 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.360620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.362675 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.362654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.369782 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.369752    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kfg\" (UniqueName: \"kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.432913 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.432868    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:46.581839 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.581805    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:06:46.583618 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:06:46.583575    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd335f9_565d_4c63_abd6_7b5b028707d6.slice/crio-9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b WatchSource:0}: Error finding container 9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b: Status 404 returned error can't find the container with id 9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b
Jul 28 14:06:46.955311 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.955269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerStarted","Data":"b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"}
Jul 28 14:06:46.955502 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:46.955343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerStarted","Data":"9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b"}
Jul 28 14:06:48.964917 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:48.964883    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/0.log"
Jul 28 14:06:48.965325 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:48.964923    2558 generic.go:356] "Generic (PLEG): container finished" podID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerID="b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c" exitCode=1
Jul 28 14:06:48.965325 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:48.965002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerDied","Data":"b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"}
Jul 28 14:06:49.576592 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:49.576543    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 28 14:06:49.971603 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:49.971567    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/0.log"
Jul 28 14:06:49.972024 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:49.971670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerStarted","Data":"b556746ca8c46b1b3a442830827b9766d9a0d93d1b8fae0608155635c44c370b"}
Jul 28 14:06:50.447619 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.447591    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:06:50.496551 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location\") pod \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") "
Jul 28 14:06:50.496551 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496555    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert\") pod \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") "
Jul 28 14:06:50.496792 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496587    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config\") pod \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") "
Jul 28 14:06:50.496792 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496632    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvwk9\" (UniqueName: \"kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9\") pod \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") "
Jul 28 14:06:50.496792 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496675    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") pod \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\" (UID: \"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1\") "
Jul 28 14:06:50.496965 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:06:50.497029 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.496978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:06:50.497029 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.497013    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:06:50.498883 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.498861    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:06:50.498962 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.498909    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9" (OuterVolumeSpecName: "kube-api-access-lvwk9") pod "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" (UID: "5cf31373-9f7f-4ea3-8523-d58b54d6d3d1"). InnerVolumeSpecName "kube-api-access-lvwk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:06:50.597614 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.597580    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:50.597614 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.597607    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvwk9\" (UniqueName: \"kubernetes.io/projected/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kube-api-access-lvwk9\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:50.597614 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.597617    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:50.597872 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.597630    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:50.597872 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.597638    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1-cabundle-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:50.978773 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.978671    2558 generic.go:356] "Generic (PLEG): container finished" podID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerID="9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653" exitCode=0
Jul 28 14:06:50.978773 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.978753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"
Jul 28 14:06:50.979312 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.978768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerDied","Data":"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"}
Jul 28 14:06:50.979312 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.978888    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f" event={"ID":"5cf31373-9f7f-4ea3-8523-d58b54d6d3d1","Type":"ContainerDied","Data":"199bb036d129df6b851417aebd620f86b1a174aff65777dd959b61d2fd7f3be2"}
Jul 28 14:06:50.979312 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.978935    2558 scope.go:117] "RemoveContainer" containerID="343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"
Jul 28 14:06:50.988322 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.988298    2558 scope.go:117] "RemoveContainer" containerID="9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"
Jul 28 14:06:50.996412 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:50.996397    2558 scope.go:117] "RemoveContainer" containerID="1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"
Jul 28 14:06:51.003949 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.003927    2558 scope.go:117] "RemoveContainer" containerID="343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"
Jul 28 14:06:51.004238 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:51.004212    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870\": container with ID starting with 343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870 not found: ID does not exist" containerID="343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"
Jul 28 14:06:51.004354 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.004250    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870"} err="failed to get container status \"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870\": rpc error: code = NotFound desc = could not find container \"343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870\": container with ID starting with 343bdda3a81ca3c8d321e2bd4e41179cf710403dea0c2667e3503670f764a870 not found: ID does not exist"
Jul 28 14:06:51.004354 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.004273    2558 scope.go:117] "RemoveContainer" containerID="9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"
Jul 28 14:06:51.004544 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:51.004525    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653\": container with ID starting with 9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653 not found: ID does not exist" containerID="9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"
Jul 28 14:06:51.004596 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.004552    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653"} err="failed to get container status \"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653\": rpc error: code = NotFound desc = could not find container \"9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653\": container with ID starting with 9ae9de7c37c81df19799a55ebc5de2bcb71f3ddf0f8719fa68a7484ab0157653 not found: ID does not exist"
Jul 28 14:06:51.004596 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.004572    2558 scope.go:117] "RemoveContainer" containerID="1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"
Jul 28 14:06:51.004942 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:51.004919    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038\": container with ID starting with 1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038 not found: ID does not exist" containerID="1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"
Jul 28 14:06:51.005041 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.004949    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038"} err="failed to get container status \"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038\": rpc error: code = NotFound desc = could not find container \"1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038\": container with ID starting with 1da1979c6e29c809bf43b76a39132f61f10a9146b528bd7629c6455a3ef18038 not found: ID does not exist"
Jul 28 14:06:51.007166 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.007137    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:06:51.009272 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:51.009254    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5fbbd668fd-drb2f"]
Jul 28 14:06:52.017396 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:52.017360    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" path="/var/lib/kubelet/pods/5cf31373-9f7f-4ea3-8523-d58b54d6d3d1/volumes"
Jul 28 14:06:56.003293 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.003263    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/1.log"
Jul 28 14:06:56.003676 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.003645    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/0.log"
Jul 28 14:06:56.003746 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.003673    2558 generic.go:356] "Generic (PLEG): container finished" podID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerID="b556746ca8c46b1b3a442830827b9766d9a0d93d1b8fae0608155635c44c370b" exitCode=1
Jul 28 14:06:56.003798 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.003755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerDied","Data":"b556746ca8c46b1b3a442830827b9766d9a0d93d1b8fae0608155635c44c370b"}
Jul 28 14:06:56.003874 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.003861    2558 scope.go:117] "RemoveContainer" containerID="b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"
Jul 28 14:06:56.004240 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.004213    2558 scope.go:117] "RemoveContainer" containerID="b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"
Jul 28 14:06:56.015089 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.015053    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_kserve-ci-e2e-test(2dd335f9-565d-4c63-abd6-7b5b028707d6)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6"
Jul 28 14:06:56.015174 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.015148    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_kserve-ci-e2e-test_2dd335f9-565d-4c63-abd6-7b5b028707d6_0 in pod sandbox 9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b from index: no such id: 'b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c'" containerID="b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"
Jul 28 14:06:56.015233 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.015180    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_kserve-ci-e2e-test_2dd335f9-565d-4c63-abd6-7b5b028707d6_0 in pod sandbox 9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b from index: no such id: 'b5c30c1c48b56cff9c1fc70a69ccf900c90817a4327f6cc957e7dd4ecf93537c'"
Jul 28 14:06:56.107954 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:56.107913    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:06:56.145857 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.145819    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 14:06:56.146035 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.145891    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls podName:2dd335f9-565d-4c63-abd6-7b5b028707d6 nodeName:}" failed. No retries permitted until 2026-07-28 14:06:56.645875216 +0000 UTC m=+5523.131499787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 14:06:56.650381 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.650344    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 14:06:56.650594 ip-10-0-138-35 kubenswrapper[2558]: E0728 14:06:56.650420    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls podName:2dd335f9-565d-4c63-abd6-7b5b028707d6 nodeName:}" failed. No retries permitted until 2026-07-28 14:06:57.650401707 +0000 UTC m=+5524.136026278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 14:06:57.009055 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.008968    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/1.log"
Jul 28 14:06:57.142633 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.142609    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/1.log"
Jul 28 14:06:57.146228 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.146211    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:57.255519 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.255481    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location\") pod \"2dd335f9-565d-4c63-abd6-7b5b028707d6\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") "
Jul 28 14:06:57.255519 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.255520    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config\") pod \"2dd335f9-565d-4c63-abd6-7b5b028707d6\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") "
Jul 28 14:06:57.255826 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.255574    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls\") pod \"2dd335f9-565d-4c63-abd6-7b5b028707d6\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") "
Jul 28 14:06:57.255826 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.255594    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5kfg\" (UniqueName: \"kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg\") pod \"2dd335f9-565d-4c63-abd6-7b5b028707d6\" (UID: \"2dd335f9-565d-4c63-abd6-7b5b028707d6\") "
Jul 28 14:06:57.255955 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.255817    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dd335f9-565d-4c63-abd6-7b5b028707d6" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:06:57.256042 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.256018    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dd335f9-565d-4c63-abd6-7b5b028707d6" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:06:57.257895 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.257872    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dd335f9-565d-4c63-abd6-7b5b028707d6" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:06:57.257943 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.257912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg" (OuterVolumeSpecName: "kube-api-access-f5kfg") pod "2dd335f9-565d-4c63-abd6-7b5b028707d6" (UID: "2dd335f9-565d-4c63-abd6-7b5b028707d6"). InnerVolumeSpecName "kube-api-access-f5kfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:06:57.356830 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.356788    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd335f9-565d-4c63-abd6-7b5b028707d6-kserve-provision-location\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:57.356830 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.356826    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:57.356830 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.356841    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd335f9-565d-4c63-abd6-7b5b028707d6-proxy-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:57.357069 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:57.356850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5kfg\" (UniqueName: \"kubernetes.io/projected/2dd335f9-565d-4c63-abd6-7b5b028707d6-kube-api-access-f5kfg\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\""
Jul 28 14:06:58.014852 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.014823    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p_2dd335f9-565d-4c63-abd6-7b5b028707d6/storage-initializer/1.log"
Jul 28 14:06:58.015263 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.014995    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"
Jul 28 14:06:58.017167 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.017139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p" event={"ID":"2dd335f9-565d-4c63-abd6-7b5b028707d6","Type":"ContainerDied","Data":"9a0c54e90975d5248dbca4ea4a852384c65b184e29263388d620319937e68a3b"}
Jul 28 14:06:58.017317 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.017201    2558 scope.go:117] "RemoveContainer" containerID="b556746ca8c46b1b3a442830827b9766d9a0d93d1b8fae0608155635c44c370b"
Jul 28 14:06:58.074913 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.074882    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:06:58.084187 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:06:58.084158    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5d56bb98fb-kvc4p"]
Jul 28 14:07:00.016886 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:00.016851    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" path="/var/lib/kubelet/pods/2dd335f9-565d-4c63-abd6-7b5b028707d6/volumes"
Jul 28 14:07:33.014512 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014473    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wskns/must-gather-h47v8"]
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014777    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014790    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014797    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014804    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014814    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kube-rbac-proxy"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014819    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kube-rbac-proxy"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014828    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014833    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014885    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014893    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kube-rbac-proxy"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014899    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf31373-9f7f-4ea3-8523-d58b54d6d3d1" containerName="kserve-container"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014949    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.014972 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014953    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.015392 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.014992    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd335f9-565d-4c63-abd6-7b5b028707d6" containerName="storage-initializer"
Jul 28 14:07:33.017798 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.017781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.021731 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.021692    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wskns\"/\"openshift-service-ca.crt\""
Jul 28 14:07:33.021882 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.021772    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wskns\"/\"kube-root-ca.crt\""
Jul 28 14:07:33.021882 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.021774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wskns\"/\"default-dockercfg-b8hlh\""
Jul 28 14:07:33.028044 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.028021    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/must-gather-h47v8"]
Jul 28 14:07:33.032783 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.032759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d28ca753-760c-40a2-aa0a-05288a360dff-must-gather-output\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.032884 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.032867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974cr\" (UniqueName: \"kubernetes.io/projected/d28ca753-760c-40a2-aa0a-05288a360dff-kube-api-access-974cr\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.133404 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.133371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d28ca753-760c-40a2-aa0a-05288a360dff-must-gather-output\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.133587 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.133445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974cr\" (UniqueName: \"kubernetes.io/projected/d28ca753-760c-40a2-aa0a-05288a360dff-kube-api-access-974cr\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.133801 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.133776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d28ca753-760c-40a2-aa0a-05288a360dff-must-gather-output\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.143484 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.143451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-974cr\" (UniqueName: \"kubernetes.io/projected/d28ca753-760c-40a2-aa0a-05288a360dff-kube-api-access-974cr\") pod \"must-gather-h47v8\" (UID: \"d28ca753-760c-40a2-aa0a-05288a360dff\") " pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.326902 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.326821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/must-gather-h47v8"
Jul 28 14:07:33.459777 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.459613    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/must-gather-h47v8"]
Jul 28 14:07:33.462516 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:07:33.462489    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd28ca753_760c_40a2_aa0a_05288a360dff.slice/crio-da22eaef1f2c9c87716df71894311289d9df01194f1cda5bf1c079eefcc8d93b WatchSource:0}: Error finding container da22eaef1f2c9c87716df71894311289d9df01194f1cda5bf1c079eefcc8d93b: Status 404 returned error can't find the container with id da22eaef1f2c9c87716df71894311289d9df01194f1cda5bf1c079eefcc8d93b
Jul 28 14:07:33.464368 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:33.464353    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:07:34.165693 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:34.165648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/must-gather-h47v8" event={"ID":"d28ca753-760c-40a2-aa0a-05288a360dff","Type":"ContainerStarted","Data":"da22eaef1f2c9c87716df71894311289d9df01194f1cda5bf1c079eefcc8d93b"}
Jul 28 14:07:35.174741 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:35.174688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/must-gather-h47v8" event={"ID":"d28ca753-760c-40a2-aa0a-05288a360dff","Type":"ContainerStarted","Data":"cc1734670d3e527ae6e791f8ea6f79041551dc9cc934a136ac116948cdc668c9"}
Jul 28 14:07:35.175182 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:35.174749    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/must-gather-h47v8" event={"ID":"d28ca753-760c-40a2-aa0a-05288a360dff","Type":"ContainerStarted","Data":"1e700856e5961e3ca880dc48ebc8379ea68f3d8b2b38782b13e48dc6b24de9ba"}
Jul 28 14:07:35.195259 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:35.195193    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wskns/must-gather-h47v8" podStartSLOduration=2.411266835 podStartE2EDuration="3.195176538s" totalImagesPullingTime="783.909703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:07:35.194684695 +0000 UTC m=+5561.680309289" watchObservedRunningTime="2026-07-28 14:07:35.195176538 +0000 UTC m=+5561.680801131"
Jul 28 14:07:36.450525 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:36.450489    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8fcfw_cdff30c9-8adc-4b6b-8e8c-a771af042897/global-pull-secret-syncer/0.log"
Jul 28 14:07:36.757092 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:36.757022    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qgx5z_bbf2d9df-0f6f-4375-9ce5-5e803a1b1f13/konnectivity-agent/0.log"
Jul 28 14:07:36.842054 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:36.842027    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-35.ec2.internal_12610501867f6e946c0cb2390ee3cc4f/haproxy/0.log"
Jul 28 14:07:41.742962 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:41.742913    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsl6j_45919c63-28b1-46cc-9a18-f774140260ba/node-exporter/0.log"
Jul 28 14:07:41.776672 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:41.776634    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsl6j_45919c63-28b1-46cc-9a18-f774140260ba/kube-rbac-proxy/0.log"
Jul 28 14:07:41.804822 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:41.804788    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsl6j_45919c63-28b1-46cc-9a18-f774140260ba/init-textfile/0.log"
Jul 28 14:07:42.378693 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:42.378655    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7c8gd_574f5331-12b8-40bc-ab0d-93ce4f00f1a4/prometheus-operator/0.log"
Jul 28 14:07:42.414548 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:42.414486    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7c8gd_574f5331-12b8-40bc-ab0d-93ce4f00f1a4/kube-rbac-proxy/0.log"
Jul 28 14:07:43.995357 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:43.995325    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-6lsv8_05b5ffc3-65b4-47c5-9e97-06f9cf9402db/networking-console-plugin/0.log"
Jul 28 14:07:44.926987 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:44.926957    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-dn64j_68125006-528f-4eb7-b4e1-9a29c9443b52/download-server/0.log"
Jul 28 14:07:46.326163 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.326076    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"]
Jul 28 14:07:46.331481 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.331453    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.342397 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.342366    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"]
Jul 28 14:07:46.440218 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.440191    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q4c6v_1f8d8022-51f4-425c-bc23-3023a521481f/dns/0.log"
Jul 28 14:07:46.462283 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.462240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7q8\" (UniqueName: \"kubernetes.io/projected/c9b14ad6-8922-4486-a4ec-7ab70920274a-kube-api-access-kl7q8\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.462477 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.462292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-podres\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.462477 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.462365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-sys\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.462477 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.462449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-lib-modules\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.462642 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.462481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-proc\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.468142 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.468119    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q4c6v_1f8d8022-51f4-425c-bc23-3023a521481f/kube-rbac-proxy/0.log"
Jul 28 14:07:46.563020 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.562980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-podres\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-sys\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-lib-modules\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-proc\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-podres\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7q8\" (UniqueName: \"kubernetes.io/projected/c9b14ad6-8922-4486-a4ec-7ab70920274a-kube-api-access-kl7q8\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563205 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-sys\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563456 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-proc\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.563456 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.563247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9b14ad6-8922-4486-a4ec-7ab70920274a-lib-modules\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.572139 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.572104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7q8\" (UniqueName: \"kubernetes.io/projected/c9b14ad6-8922-4486-a4ec-7ab70920274a-kube-api-access-kl7q8\") pod \"perf-node-gather-daemonset-5ww66\" (UID: \"c9b14ad6-8922-4486-a4ec-7ab70920274a\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.599880 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.599848    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sq4xr_adff46d3-a8d6-44ec-bb7b-00a01a52ca78/dns-node-resolver/0.log"
Jul 28 14:07:46.644871 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.644819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:46.806574 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:46.806406    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"]
Jul 28 14:07:46.812687 ip-10-0-138-35 kubenswrapper[2558]: W0728 14:07:46.811124    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9b14ad6_8922_4486_a4ec_7ab70920274a.slice/crio-68973eb0ca86c87edfd12e3fa357e78b72859d41e3cf11791aa0be8dbdaacad7 WatchSource:0}: Error finding container 68973eb0ca86c87edfd12e3fa357e78b72859d41e3cf11791aa0be8dbdaacad7: Status 404 returned error can't find the container with id 68973eb0ca86c87edfd12e3fa357e78b72859d41e3cf11791aa0be8dbdaacad7
Jul 28 14:07:47.123362 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.123276    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-585cb7d774-wgzt6_cb90edc5-d7a7-4915-9e67-5246cdc52234/registry/0.log"
Jul 28 14:07:47.244213 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.244177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66" event={"ID":"c9b14ad6-8922-4486-a4ec-7ab70920274a","Type":"ContainerStarted","Data":"c11dd900e097d117cfa3f98ec387ec32106fd2a240cfe393c0701071c12cf11e"}
Jul 28 14:07:47.244397 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.244239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66" event={"ID":"c9b14ad6-8922-4486-a4ec-7ab70920274a","Type":"ContainerStarted","Data":"68973eb0ca86c87edfd12e3fa357e78b72859d41e3cf11791aa0be8dbdaacad7"}
Jul 28 14:07:47.244397 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.244267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:47.282243 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.282187    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66" podStartSLOduration=1.28217288 podStartE2EDuration="1.28217288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:07:47.2791701 +0000 UTC m=+5573.764794692" watchObservedRunningTime="2026-07-28 14:07:47.28217288 +0000 UTC m=+5573.767797483"
Jul 28 14:07:47.288198 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:47.288157    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t724j_8defa946-39f1-480d-9c8c-60855d3164c0/node-ca/0.log"
Jul 28 14:07:48.490265 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:48.490237    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m9zjd_cc234e4e-0c55-4983-bbb0-e129c1b63107/serve-healthcheck-canary/0.log"
Jul 28 14:07:49.137515 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:49.137476    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfppf_90d16b4e-aa12-40c5-a466-2202e6a14ff3/kube-rbac-proxy/0.log"
Jul 28 14:07:49.170322 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:49.170287    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfppf_90d16b4e-aa12-40c5-a466-2202e6a14ff3/exporter/0.log"
Jul 28 14:07:49.200090 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:49.200063    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfppf_90d16b4e-aa12-40c5-a466-2202e6a14ff3/extractor/0.log"
Jul 28 14:07:51.645679 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:51.645647    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-5l87j_953869a9-7494-4843-b179-fb4f94bf1dae/server/0.log"
Jul 28 14:07:51.975215 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:51.975121    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-98zz9_11cba7c0-d129-4c6c-8737-12be323fa510/manager/0.log"
Jul 28 14:07:52.155620 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:52.155588    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-cr5ch_c0990dfe-051a-4be5-9fdc-c48654f75693/seaweedfs/0.log"
Jul 28 14:07:52.183252 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:52.183228    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-8vqsp_4b9bd650-b867-40e6-8811-042ae7f62a4c/seaweedfs-tls-custom/0.log"
Jul 28 14:07:52.210779 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:52.210746    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-h8mrh_5aacb345-2e47-49c6-be35-16d7455d797d/seaweedfs-tls-serving/0.log"
Jul 28 14:07:53.264135 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:53.264104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-5ww66"
Jul 28 14:07:56.457934 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:56.457908    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-bt9cn_d6979445-3b85-4f61-a570-b21e8b788ffd/migrator/0.log"
Jul 28 14:07:56.490500 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:56.490472    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-bt9cn_d6979445-3b85-4f61-a570-b21e8b788ffd/graceful-termination/0.log"
Jul 28 14:07:58.263199 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.263173    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/kube-multus-additional-cni-plugins/0.log"
Jul 28 14:07:58.310580 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.310484    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/egress-router-binary-copy/0.log"
Jul 28 14:07:58.356879 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.356847    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/cni-plugins/0.log"
Jul 28 14:07:58.400480 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.400455    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/bond-cni-plugin/0.log"
Jul 28 14:07:58.441579 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.441554    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/routeoverride-cni/0.log"
Jul 28 14:07:58.506422 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.506396    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/whereabouts-cni-bincopy/0.log"
Jul 28 14:07:58.551965 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:58.551935    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6z5b_57be213e-d569-49c8-8f8d-39e47e357423/whereabouts-cni/0.log"
Jul 28 14:07:59.038464 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:59.038434    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fbc2s_ed418fcc-cb62-48f8-a809-6e8db21811fd/kube-multus/0.log"
Jul 28 14:07:59.243933 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:59.243905    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qhblp_911090a5-d1b2-49eb-ab42-3a239ccaa4f0/network-metrics-daemon/0.log"
Jul 28 14:07:59.272851 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:07:59.272822    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qhblp_911090a5-d1b2-49eb-ab42-3a239ccaa4f0/kube-rbac-proxy/0.log"
Jul 28 14:08:00.313451 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.313411    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-controller/0.log"
Jul 28 14:08:00.336127 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.336097    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/1.log"
Jul 28 14:08:00.336127 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.336114    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovn-acl-logging/0.log"
Jul 28 14:08:00.356886 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.356856    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/kube-rbac-proxy-node/0.log"
Jul 28 14:08:00.381686 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.381653    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 14:08:00.406819 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.406793    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/northd/0.log"
Jul 28 14:08:00.433876 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.433852    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/nbdb/0.log"
Jul 28 14:08:00.461816 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.461778    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/sbdb/0.log"
Jul 28 14:08:00.523263 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:00.523226    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k48sz_211840ec-bc0a-42ca-9b18-7feef5d047b9/ovnkube-controller/0.log"
Jul 28 14:08:01.771554 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:01.771520    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-dq99c_eddedb54-cc14-4cb7-ac99-4683df900722/check-endpoints/0.log"
Jul 28 14:08:01.810263 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:01.810233    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5cmjc_611e17b6-2857-4c4d-93e1-77d997af210d/network-check-target-container/0.log"
Jul 28 14:08:02.836171 ip-10-0-138-35 kubenswrapper[2558]: I0728 14:08:02.836140    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fw29v_ccdb8e52-a2d9-4925-857e-51be064d7650/iptables-alerter/0.log"