Jul 28 17:15:03.077053 ip-10-0-138-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:15:03.077063 ip-10-0-138-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:15:03.077070 ip-10-0-138-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:15:03.077332 ip-10-0-138-92 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:15:13.280136 ip-10-0-138-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:15:13.280155 ip-10-0-138-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9dd2b81927744eaf8494d8f7e53af477 --
Jul 28 17:17:09.666150 ip-10-0-138-92 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:17:10.027560 ip-10-0-138-92 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:10.027560 ip-10-0-138-92 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:17:10.027560 ip-10-0-138-92 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:10.027560 ip-10-0-138-92 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:17:10.027560 ip-10-0-138-92 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:10.028412 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.028326    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:17:10.033210 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033193    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.033210 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033210    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033214    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033218    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033221    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033226    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033228    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033231    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033234    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033237    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033239    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033242    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033246    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033250    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033253    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033256    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033259    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033261    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033264    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033267    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.033281 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033269    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033272    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033277    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033281    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033284    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033287    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033290    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033292    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033295    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033298    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033300    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033303    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033306    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033308    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033311    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033313    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033316    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033318    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033320    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033323    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.033775 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033325    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033328    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033330    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033333    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033335    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033338    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033341    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033343    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033347    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033349    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033352    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033355    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033357    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033360    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033363    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033365    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033368    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033371    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033373    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033375    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.034280 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033378    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033381    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033383    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033386    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033393    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033396    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033400    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033403    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033406    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033408    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033411    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033414    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033417    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033419    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033422    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033425    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033427    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033430    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033433    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.034766 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033437    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033439    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033442    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033444    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033447    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033449    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033453    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033456    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033458    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033461    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033463    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033466    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033469    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033471    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033473    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033476    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033478    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033481    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033487    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.035343 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.033490    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.037532 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037514    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.037532 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037531    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037535    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037539    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037542    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037546    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037549    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037552    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037555    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037559    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037561    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037564    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037567    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037570    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037572    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037575    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037578    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037581    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037584    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037587    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037589    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.037606 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037592    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037595    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037597    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037601    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037606    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037609    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037612    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037616    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037620    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037623    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037626    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037631    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037635    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037638    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037641    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037643    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037646    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037649    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037651    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.038129 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037654    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037657    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037659    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037662    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037665    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037667    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037670    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037673    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037675    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037678    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037680    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037683    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037685    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037688    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037691    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037695    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037698    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037701    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037703    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037706    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.038595 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037710    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037713    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037716    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037719    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037721    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037724    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037727    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037730    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037732    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037735    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037737    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037740    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037743    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037745    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037748    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037751    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037753    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037756    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037758    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037761    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.039107 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037764    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037766    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037769    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037773    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037778    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037781    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037784    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037788    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037791    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037794    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037796    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037799    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037802    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037804    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037807    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037810    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037812    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037815    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.037817    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038243    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.039623 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038249    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038251    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038254    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038257    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038260    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038263    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038265    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038269    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038274    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038276    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038279    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038282    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038284    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038286    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038289    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038292    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038295    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038298    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038300    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.040136 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038303    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038306    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038308    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038311    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038313    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038316    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038318    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038321    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038323    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038326    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038328    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038331    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038335    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038339    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038342    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038345    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038348    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038351    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038353    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.040661 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038356    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038358    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038361    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038363    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038366    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038368    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038371    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038373    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038377    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038379    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038382    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038384    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038387    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038392    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038395    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038397    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038400    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038403    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038406    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038409    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.041147 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038411    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038414    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038416    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038418    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038421    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038424    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038426    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038429    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038431    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038434    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038436    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038438    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038441    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038444    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038446    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038449    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038452    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038456    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038459    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038462    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.041634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038465    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038468    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038471    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038474    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038476    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038479    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038482    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038484    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038487    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038489    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038492    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038495    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038497    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038499    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038502    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038504    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038507    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038509    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.038512    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038586    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:17:10.042177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038593    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038600    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038605    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038609    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038613    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038618    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038622    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038626    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038629    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038632    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038636    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038639    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038642    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038644    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038647    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038650    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038653    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038657    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038660    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038663    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038666    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038669    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038673    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:17:10.042680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038676    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038679    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038682    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038685    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038688    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038691    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038694    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038697    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038702    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038704    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038707    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038710    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038713    2561 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038716    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038721    2561 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038724    2561 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038727    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038731    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038733    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038737    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038740    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038743    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038746    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038749    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038751    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:17:10.043248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038754    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038757    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038760    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038763    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038766    2561 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038776    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038780    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038783    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038787    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038790    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038793    2561 flags.go:64] FLAG: --help="false"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038796    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-92.ec2.internal"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038799    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038802    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038805    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038809    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038812    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038815    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038818    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038821    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038824    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038842    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038845    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038851    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:17:10.043904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038854    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038857    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038860    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038863    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038866    2561 flags.go:64] FLAG: --lock-file=""
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038869    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038872    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038875    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038881    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038884    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038887    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038890    2561 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038892    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038896    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038899    2561 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038901    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038906    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038924    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038929    2561 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038933    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038936    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038940    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038943    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038946    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038949    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:17:10.044495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038952    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038960    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038963    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038966    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038968    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038971    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038978    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038981    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038984    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038987    2561 flags.go:64] FLAG: --port="10250"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038990    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038993    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02fddcda5f077079a"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038996    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.038999    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039002    2561 flags.go:64] FLAG: --register-node="true"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039005    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039008    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039011    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039014    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039017    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039020    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039023    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039026    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039029    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039032    2561 flags.go:64] FLAG: --runonce="false"
Jul 28 17:17:10.045466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039034    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039037    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039040    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039043    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039049    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039052    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039055    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039058    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039061    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039064    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039067    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039070    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039073    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039078    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039081    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039086    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039089    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039092    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039096    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039098    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039101    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039104    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039107    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039110    2561 flags.go:64] FLAG: --v="2"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039114    2561 flags.go:64] FLAG: --version="false"
Jul 28 17:17:10.046504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039119    2561 flags.go:64] FLAG: --vmodule=""
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039122    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039126    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039242    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039245    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039248    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039251    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039254    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039256    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039259    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039262    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039265    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039268    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039271    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039274    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039276    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039279    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039282    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039285    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.047194 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039289    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039294    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039297    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039300    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039302    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039305    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039308    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039310    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039313    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039315    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039320    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039323    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039326    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039329    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039332    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039334    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039337    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039339    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039342    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.047677 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039344    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039347    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039349    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039352    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039355    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039359    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039362    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039364    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039367    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039370    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039372    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039376    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039378    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039381    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039385    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039388    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039390    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039393    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039396    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039398    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.048254 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039401    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039403    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039406    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039409    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039411    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039414    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039416    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039418    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039422    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039424    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039427    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039429    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039432    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039434    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039437    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039439    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039442    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039445    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039448    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039451    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.048748 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039453    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039456    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039458    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039461    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039464    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039467    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039471    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039473    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039476    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039479    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039481    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039484    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039486    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039489    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039491    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039494    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039496    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039499    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039502    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.049242 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039504    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039508    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039511    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039514    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.039517    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.039522    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.047515    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.047534    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.047555    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.047561    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047620    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047625    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047628    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047632    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.049711 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047635    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047638    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047641    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047644    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047646    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047649    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047651    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047655    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047657    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047660    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047662    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047665    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047668    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047671    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047674    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047676    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047679    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047681    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047684    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047687    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.050074 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047690    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047693    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047695    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047699    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047703    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047707    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047710    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047713    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047716    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047719    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047721    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047724    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047728    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047732    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047735    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047737    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047740    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047743    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047745    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.050576 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047747    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047751    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047754    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047758    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047761    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047764    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047767    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047770    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047773    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047775    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047778    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047781    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047784    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047787    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047790    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047792    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047795    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047798    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047802    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047804    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.051069 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047807    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047810    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047813    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047815    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047818    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047820    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047823    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047843    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047846    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047849    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047852    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047854    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047857    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047860    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047863    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047865    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047869    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047872    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047874    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047877    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.051563 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047879    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047882    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047885    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047887    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047890    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047892    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047895    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047898    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047900    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047903    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047906    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047909    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047911    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047914    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047916    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.052141 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.047919    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.047924    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048047    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048051    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048054    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048057    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048060    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048063    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048065    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048068    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048071    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048073    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048076    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048078    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048081    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:10.052509 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048084    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048087    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048091    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048094    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048097    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048101    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048104    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048107    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048110    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048113    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048116    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048119    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048121    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048124    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048126    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048129    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048132    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048134    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048137    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048139    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:10.052890 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048142    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048144    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048147    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048149    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048151    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048154    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048156    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048159    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048161    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048164    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048166    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048169    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048172    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048174    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048177    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048180    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048182    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048185    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048187    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048190    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:10.053377 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048192    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048195    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048198    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048200    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048203    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048205    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048208    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048210    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048213    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048215    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048218    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048220    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048223    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048225    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048228    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048230    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048234    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048237    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048240    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:10.053906 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048243    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048245    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048248    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048250    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048253    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048255    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048259    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048261    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048264    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048267    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048269    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048272    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048275    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048277    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048279    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048282    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048285    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048287    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048289    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048292    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:10.054361 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048294    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048297    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048300    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048304    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048307    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048310    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:10.048313    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.048318    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.048439    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.051399    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.052855    2561 server.go:1019] "Starting client certificate rotation"
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.052952    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:10.054886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.052987    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:10.055215 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.055049    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:10.056867 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.056851    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:10.066316 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.066301    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:17:10.068656 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.068642    2561 log.go:25] "Validated CRI v1 image API"
Jul 28 17:17:10.071878 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.071858    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:17:10.074365 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.074347    2561 fs.go:135] Filesystem UUIDs: map[073e2b15-9195-4a69-bbde-a09b0b103a05:/dev/nvme0n1p3 16a97317-155b-468c-a504-81f3cf301461:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:17:10.074445 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.074364    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:17:10.079312 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.079203    2561 manager.go:216] Machine: {Timestamp:2026-07-28 17:17:10.078708745 +0000 UTC m=+0.303624072 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3149322 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238f3037e3293bde05baac649db8e4 SystemUUID:ec238f30-37e3-293b-de05-baac649db8e4 BootID:9dd2b819-2774-4eaf-8494-d8f7e53af477 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:60:f5:7d:66:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:60:f5:7d:66:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:5f:64:ee:bb:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:17:10.079312 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.079308    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:17:10.079465 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.079381    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:17:10.084023 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.083996    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:17:10.084184 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.084024    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:17:10.084840 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.084816    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:17:10.084884 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.084846    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:17:10.084884 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.084853    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:10.084884 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.084869    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:10.085803 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.085791    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:17:10.087582 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087571    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:10.087714 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087704    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:17:10.087747 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087733    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:17:10.087747 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087744    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:17:10.087801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087764    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:17:10.087801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087773    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:17:10.087801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087783    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:17:10.087801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.087782    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:10.088193 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.088181    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:10.088232 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.088200    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:10.090320 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.090301    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:17:10.092071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.092057    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:17:10.092126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.092087    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:17:10.093425 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093413    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093429    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093435    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093441    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093450    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093458    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:17:10.093469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093469    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:17:10.093635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093475    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:17:10.093635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093482    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:17:10.093635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093487    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:17:10.093635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093495    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:17:10.093635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.093500    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:17:10.094860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.094848    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:17:10.094933 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.094883    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:17:10.097809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.097797    2561 server.go:1268] "Started kubelet"
Jul 28 17:17:10.098399 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.098368    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:17:10.098627 ip-10-0-138-92 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:17:10.098817 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.098724    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:17:10.098817 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.098807    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:17:10.100425 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.100406    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:17:10.100854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.100814    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:17:10.102584 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.102564    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:17:10.104494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.104475    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:17:10.104583 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.104565    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:17:10.104633 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.104538    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:17:10.105387 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.104522    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-92.ec2.internal.18c683a1110931bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-92.ec2.internal,UID:ip-10-0-138-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-92.ec2.internal,},FirstTimestamp:2026-07-28 17:17:10.097768895 +0000 UTC m=+0.322684223,LastTimestamp:2026-07-28 17:17:10.097768895 +0000 UTC m=+0.322684223,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-92.ec2.internal,}"
Jul 28 17:17:10.106151 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106131    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:17:10.106151 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106151    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:10.106303 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.106202    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:10.106875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106850    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:17:10.107004 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106881    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:17:10.107004 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106894    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:17:10.107004 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106979    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:17:10.107004 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.106989    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:17:10.107175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107082    2561 factory.go:153] Registering CRI-O factory
Jul 28 17:17:10.107175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107102    2561 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:17:10.107235 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107190    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:17:10.107235 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107202    2561 factory.go:55] Registering systemd factory
Jul 28 17:17:10.107235 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107212    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:17:10.107322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107238    2561 factory.go:103] Registering Raw factory
Jul 28 17:17:10.107322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107256    2561 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:17:10.107892 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.107880    2561 manager.go:349] Starting recovery of all containers
Jul 28 17:17:10.108710 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.108660    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:17:10.108800 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.108758    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:17:10.119442 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.119425    2561 manager.go:354] Recovery completed
Jul 28 17:17:10.123688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.123592    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:10.125792 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.125777    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:10.125908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.125806    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:10.125908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.125816    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:10.126429 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.126413    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:17:10.126429 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.126426    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:17:10.126524 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.126439    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:10.126524 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.126468    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:10.128011 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.127999    2561 policy_none.go:49] "None policy: Start"
Jul 28 17:17:10.128051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.128014    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:17:10.128051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.128023    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:17:10.128489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.128473    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:10.128544 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.128503    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:10.128544 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.128518    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:10.129090 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.129077    2561 policy_none.go:47] "Start"
Jul 28 17:17:10.138541 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.137718    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-92.ec2.internal.18c683a112b4cea8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-92.ec2.internal,UID:ip-10-0-138-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-92.ec2.internal,},FirstTimestamp:2026-07-28 17:17:10.125792936 +0000 UTC m=+0.350708266,LastTimestamp:2026-07-28 17:17:10.125792936 +0000 UTC m=+0.350708266,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-92.ec2.internal,}"
Jul 28 17:17:10.139458 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.139441    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7kcq"
Jul 28 17:17:10.151598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.151573    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7kcq"
Jul 28 17:17:10.153675 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.153586    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-92.ec2.internal.18c683a112b51476  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-92.ec2.internal,UID:ip-10-0-138-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-92.ec2.internal,},FirstTimestamp:2026-07-28 17:17:10.125810806 +0000 UTC m=+0.350726134,LastTimestamp:2026-07-28 17:17:10.125810806 +0000 UTC m=+0.350726134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-92.ec2.internal,}"
Jul 28 17:17:10.156077 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.156015    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-92.ec2.internal.18c683a112b53bd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-92.ec2.internal,UID:ip-10-0-138-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-92.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-92.ec2.internal,},FirstTimestamp:2026-07-28 17:17:10.125820883 +0000 UTC m=+0.350736211,LastTimestamp:2026-07-28 17:17:10.125820883 +0000 UTC m=+0.350736211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-92.ec2.internal,}"
Jul 28 17:17:10.174495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.174474    2561 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:17:10.174616 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.174534    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:17:10.174616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.174549    2561 server.go:87] "Starting device plugin registration server"
Jul 28 17:17:10.174865 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.174850    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:17:10.174940 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.174867    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:17:10.174993 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.174979    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:17:10.175092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.175074    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:17:10.175092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:10.175088    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:17:10.175843 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.175812    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:17:10.175926 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:10.175884    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.183009 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.182978    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:12.184079 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.184060    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:12.184162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.184101    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:12.184162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.184113    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:12.184162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.184138    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.194360 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.194294    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.194360 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.194332    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-92.ec2.internal\": node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.196545 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.196510    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:17:12.197714 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.197695    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:17:12.197804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.197734    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:17:12.197804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.197755    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:17:12.197804 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.197791    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:17:12.199969 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.199947    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:12.211318 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.211279    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.298430 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.298392    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"]
Jul 28 17:17:12.298552 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.298496    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:12.300114 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.300095    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:12.300216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.300132    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:12.300216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.300146    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:12.301391 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.301376    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:12.301496 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.301479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.301551 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.301511    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:12.302204 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302189    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:12.302276 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302216    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:12.302276 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302230    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:12.302760 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302748    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:12.302834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302772    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:12.302834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.302786    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:12.303879 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.303861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.303969 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.303885    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:12.304598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.304582    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:12.304668 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.304610    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:12.304668 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.304624    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:12.311589 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.311570    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.316371 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.316356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.316440 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.316378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.316440 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.316394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e04dce7c55a72020d01171561fa4ac33-config\") pod \"kube-apiserver-proxy-ip-10-0-138-92.ec2.internal\" (UID: \"e04dce7c55a72020d01171561fa4ac33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.325940 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.325927    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-92.ec2.internal\" not found" node="ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.330554 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.330537    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-92.ec2.internal\" not found" node="ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.412985 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.411747    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.417364 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.417488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.417488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e04dce7c55a72020d01171561fa4ac33-config\") pod \"kube-apiserver-proxy-ip-10-0-138-92.ec2.internal\" (UID: \"e04dce7c55a72020d01171561fa4ac33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.417488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e04dce7c55a72020d01171561fa4ac33-config\") pod \"kube-apiserver-proxy-ip-10-0-138-92.ec2.internal\" (UID: \"e04dce7c55a72020d01171561fa4ac33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.417488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.417619 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.417495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1558a3aa7615c15900b8d30d171fa55d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal\" (UID: \"1558a3aa7615c15900b8d30d171fa55d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.512859 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.512777    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.613355 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.613294    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.628700 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.628676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.633701 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.633684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:12.713864 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.713821    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.814360 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.814272    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.914744 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:12.914708    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:12.960195 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.960157    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:17:12.960772 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:12.960392    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:17:13.015400 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:13.015369    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:13.064593 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.064507    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:12:12 +0000 UTC" deadline="2028-05-02 06:04:34.23221925 +0000 UTC"
Jul 28 17:17:13.064593 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.064542    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15444h47m21.167680961s"
Jul 28 17:17:13.081050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.081024    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:13.116319 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:13.116283    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:13.216926 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:13.216893    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-92.ec2.internal\" not found"
Jul 28 17:17:13.241357 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:13.241324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1558a3aa7615c15900b8d30d171fa55d.slice/crio-2057881804bda05763f37dcb3d567c42602b14f3e4be388072b7814e9a64dd25 WatchSource:0}: Error finding container 2057881804bda05763f37dcb3d567c42602b14f3e4be388072b7814e9a64dd25: Status 404 returned error can't find the container with id 2057881804bda05763f37dcb3d567c42602b14f3e4be388072b7814e9a64dd25
Jul 28 17:17:13.241687 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:13.241659    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04dce7c55a72020d01171561fa4ac33.slice/crio-fa26bdf1b02b17797b93f75912b18dec2cbb9135852324912089f1d3e5eecf94 WatchSource:0}: Error finding container fa26bdf1b02b17797b93f75912b18dec2cbb9135852324912089f1d3e5eecf94: Status 404 returned error can't find the container with id fa26bdf1b02b17797b93f75912b18dec2cbb9135852324912089f1d3e5eecf94
Jul 28 17:17:13.246056 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.246041    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:17:13.258793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.258773    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:13.260123 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.260103    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:13.260209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.260135    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:13.260209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.260146    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:13.285126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.285099    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:13.314193 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.314160    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:13.326689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.326630    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:13.328244 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.328230    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal"
Jul 28 17:17:13.336714 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.336691    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:13.571046 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.571019    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:13.831501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.831473    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:13.995855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:13.995809    2561 apiserver.go:52] "Watching apiserver"
Jul 28 17:17:14.006035 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.006007    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:17:14.006501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.006472    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-vgb77","kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m","openshift-multus/multus-2m8g7","openshift-multus/multus-additional-cni-plugins-4qw6b","openshift-network-diagnostics/network-check-target-sck52","openshift-ovn-kubernetes/ovnkube-node-6spk5","kube-system/global-pull-secret-syncer-rw72f","kube-system/konnectivity-agent-rcqhx","openshift-cluster-node-tuning-operator/tuned-xzrm9","openshift-dns/node-resolver-dmvp4","openshift-image-registry/node-ca-knslj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal","openshift-multus/network-metrics-daemon-7nphk"]
Jul 28 17:17:14.009372 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.009350    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.011548 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.011527    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.013562 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.013542    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.015224 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.014329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.015224 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.014599    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:14.016959 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.016914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.019292 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.019046    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.019815 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.019796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.020209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:17:14.020291 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020263    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk4jv\""
Jul 28 17:17:14.020794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020440    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sjjq2\""
Jul 28 17:17:14.020794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020665    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:17:14.020794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020683    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.020794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9rl\""
Jul 28 17:17:14.020794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020787    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:17:14.021088 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.020826    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.021088 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.021026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:17:14.021546 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.021215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fffnx\""
Jul 28 17:17:14.021546 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.021522    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.021772 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.021697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:17:14.022351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.022131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:14.022440 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.022360    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.022775 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.022591    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:14.024579 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.024556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:17:14.024809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.024794    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:17:14.025489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.025489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.025489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-log-socket\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-registration-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-multus-certs\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-systemd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-var-lib-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-ovn\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.025694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-socket-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.025976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wfl\" (UniqueName: \"kubernetes.io/projected/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kube-api-access-l6wfl\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.025976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-system-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.025976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-netns\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.025976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-netns\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.025953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-bin\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/5befc373-0bba-4238-839c-7463d45b4913-kube-api-access-2nr6t\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvw5\" (UniqueName: \"kubernetes.io/projected/ecb1f853-e9c4-4d6a-8540-f76de0162149-kube-api-access-zvvw5\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.026331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-cnibin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-os-release\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-kubelet\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-script-lib\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-multus\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kft8m\" (UniqueName: \"kubernetes.io/projected/6cadb470-ea44-4d68-ba72-673593f2f908-kube-api-access-kft8m\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.026576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-cni-binary-copy\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-socket-dir-parent\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-bin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-kubelet\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-env-overrides\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-conf-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-multus-daemon-config\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-systemd-units\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-slash\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.026807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5befc373-0bba-4238-839c-7463d45b4913-ovn-node-metrics-cert\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-etc-kubernetes\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.026983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-etc-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-node-log\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-netd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-config\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecb1f853-e9c4-4d6a-8540-f76de0162149-hosts-file\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb1f853-e9c4-4d6a-8540-f76de0162149-tmp-dir\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-device-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-sys-fs\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-k8s-cni-cncf-io\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.027566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.027292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-hostroot\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.028347 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.028328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.028448 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.028353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.028448 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.028413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:14.028562 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.028487    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.029660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2q9s9\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.029741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.029858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.029894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x2pz5\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.029925    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.030841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.030224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:17:14.032072 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.032052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.032166 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.032131    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:14.035996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.035174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.037218 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.037199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.038216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.038195    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:14.038769 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.038753    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.038856 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.038830    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.039123 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.039105    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wwlbx\""
Jul 28 17:17:14.039628 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.039608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.040156 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.040124    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.040239 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.040181    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:17:14.040239 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.040195    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.040456 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.040384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vdc57\""
Jul 28 17:17:14.041971 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.041948    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:17:14.042203 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.042186    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.042364 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.042347    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.042428 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.042419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87wbm\""
Jul 28 17:17:14.053510 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.053486    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-87g72"
Jul 28 17:17:14.069676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.069659    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-87g72"
Jul 28 17:17:14.116075 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.116050    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:17:14.127597 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-multus\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-kubelet-config\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-modprobe-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-multus\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-socket-dir-parent\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-env-overrides\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.127768 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-socket-dir-parent\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-systemd\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-host\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-conf-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-systemd-units\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-node-log\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-netd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-conf-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.127962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-config\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-sys\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82531788-de8e-41d8-92ca-ec35a6e2521d-host-slash\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-k8s-cni-cncf-io\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-hostroot\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-cnibin\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-node-log\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-k8s-cni-cncf-io\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjrf\" (UniqueName: \"kubernetes.io/projected/7bcaf0bd-eabc-4f4f-a938-83f807904b03-kube-api-access-7pjrf\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-registration-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-hostroot\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whr9\" (UniqueName: \"kubernetes.io/projected/82531788-de8e-41d8-92ca-ec35a6e2521d-kube-api-access-5whr9\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-env-overrides\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-socket-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-system-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-socket-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-netns\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-netns\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-netns\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-bin\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-system-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.128860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/5befc373-0bba-4238-839c-7463d45b4913-kube-api-access-2nr6t\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-netd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-os-release\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-cni-bin\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-kubernetes\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-netns\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-config\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-systemd-units\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-registration-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-cnibin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9w44\" (UniqueName: \"kubernetes.io/projected/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-kube-api-access-h9w44\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-cnibin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.128978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bcaf0bd-eabc-4f4f-a938-83f807904b03-host\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kft8m\" (UniqueName: \"kubernetes.io/projected/6cadb470-ea44-4d68-ba72-673593f2f908-kube-api-access-kft8m\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtxt\" (UniqueName: \"kubernetes.io/projected/13a33bf6-6706-40de-bdb3-a635e075db13-kube-api-access-5jtxt\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.129644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9edf578f-d23b-45c5-81b6-582b69a24d5f-konnectivity-ca\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysconfig\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-cni-binary-copy\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-bin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-kubelet\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-tmp\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bcaf0bd-eabc-4f4f-a938-83f807904b03-serviceca\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-cni-bin\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-multus-daemon-config\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-kubelet\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-slash\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5befc373-0bba-4238-839c-7463d45b4913-ovn-node-metrics-cert\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.130455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-etc-kubernetes\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-etc-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-cni-binary-copy\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9edf578f-d23b-45c5-81b6-582b69a24d5f-agent-certs\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-run\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cadb470-ea44-4d68-ba72-673593f2f908-multus-daemon-config\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-etc-kubernetes\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82531788-de8e-41d8-92ca-ec35a6e2521d-iptables-alerter-script\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-etc-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecb1f853-e9c4-4d6a-8540-f76de0162149-hosts-file\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb1f853-e9c4-4d6a-8540-f76de0162149-tmp-dir\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-device-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.129998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-sys-fs\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130006    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-log-socket\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-multus-certs\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-systemd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-var-lib-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-ovn\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb1f853-e9c4-4d6a-8540-f76de0162149-tmp-dir\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-dbus\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-run-multus-certs\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-conf\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wfl\" (UniqueName: \"kubernetes.io/projected/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kube-api-access-l6wfl\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdll\" (UniqueName: \"kubernetes.io/projected/5ef737e1-0408-47c9-962c-ac8ad414f763-kube-api-access-smdll\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-multus-cni-dir\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-ovn\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ecb1f853-e9c4-4d6a-8540-f76de0162149-hosts-file\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-lib-modules\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-device-dir\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.131955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-var-lib-kubelet\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1989af2e-a938-451b-a1bf-f5f91e5a07cc-sys-fs\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130585    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-log-socket\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-run-systemd\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-var-lib-openvswitch\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-slash\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5befc373-0bba-4238-839c-7463d45b4913-host-run-ovn-kubernetes\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvw5\" (UniqueName: \"kubernetes.io/projected/ecb1f853-e9c4-4d6a-8540-f76de0162149-kube-api-access-zvvw5\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.130796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-os-release\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-kubelet\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-os-release\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-script-lib\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-system-cni-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cadb470-ea44-4d68-ba72-673593f2f908-host-var-lib-kubelet\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.132756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-tuned\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.133567 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.131708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5befc373-0bba-4238-839c-7463d45b4913-ovnkube-script-lib\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.133567 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.133480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5befc373-0bba-4238-839c-7463d45b4913-ovn-node-metrics-cert\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.138433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.137825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/5befc373-0bba-4238-839c-7463d45b4913-kube-api-access-2nr6t\") pod \"ovnkube-node-6spk5\" (UID: \"5befc373-0bba-4238-839c-7463d45b4913\") " pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.138433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.138025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kft8m\" (UniqueName: \"kubernetes.io/projected/6cadb470-ea44-4d68-ba72-673593f2f908-kube-api-access-kft8m\") pod \"multus-2m8g7\" (UID: \"6cadb470-ea44-4d68-ba72-673593f2f908\") " pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.143413 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.143387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvw5\" (UniqueName: \"kubernetes.io/projected/ecb1f853-e9c4-4d6a-8540-f76de0162149-kube-api-access-zvvw5\") pod \"node-resolver-dmvp4\" (UID: \"ecb1f853-e9c4-4d6a-8540-f76de0162149\") " pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.146016 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.145972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wfl\" (UniqueName: \"kubernetes.io/projected/1989af2e-a938-451b-a1bf-f5f91e5a07cc-kube-api-access-l6wfl\") pod \"aws-ebs-csi-driver-node-vnc4m\" (UID: \"1989af2e-a938-451b-a1bf-f5f91e5a07cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.202472 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.202414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal" event={"ID":"e04dce7c55a72020d01171561fa4ac33","Type":"ContainerStarted","Data":"fa26bdf1b02b17797b93f75912b18dec2cbb9135852324912089f1d3e5eecf94"}
Jul 28 17:17:14.203682 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.203653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal" event={"ID":"1558a3aa7615c15900b8d30d171fa55d","Type":"ContainerStarted","Data":"2057881804bda05763f37dcb3d567c42602b14f3e4be388072b7814e9a64dd25"}
Jul 28 17:17:14.232234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-kubernetes\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.232501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9w44\" (UniqueName: \"kubernetes.io/projected/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-kube-api-access-h9w44\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bcaf0bd-eabc-4f4f-a938-83f807904b03-host\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.232501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtxt\" (UniqueName: \"kubernetes.io/projected/13a33bf6-6706-40de-bdb3-a635e075db13-kube-api-access-5jtxt\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9edf578f-d23b-45c5-81b6-582b69a24d5f-konnectivity-ca\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysconfig\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-tmp\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bcaf0bd-eabc-4f4f-a938-83f807904b03-serviceca\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9edf578f-d23b-45c5-81b6-582b69a24d5f-agent-certs\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-run\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82531788-de8e-41d8-92ca-ec35a6e2521d-iptables-alerter-script\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.232705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-dbus\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-conf\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysconfig\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdll\" (UniqueName: \"kubernetes.io/projected/5ef737e1-0408-47c9-962c-ac8ad414f763-kube-api-access-smdll\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-lib-modules\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-conf\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-var-lib-kubelet\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-run\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-system-cni-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-system-cni-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.232980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-tuned\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-kubelet-config\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-modprobe-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-systemd\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-host\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-sys\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82531788-de8e-41d8-92ca-ec35a6e2521d-host-slash\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-cnibin\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjrf\" (UniqueName: \"kubernetes.io/projected/7bcaf0bd-eabc-4f4f-a938-83f807904b03-kube-api-access-7pjrf\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whr9\" (UniqueName: \"kubernetes.io/projected/82531788-de8e-41d8-92ca-ec35a6e2521d-kube-api-access-5whr9\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-os-release\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-os-release\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-kubernetes\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bcaf0bd-eabc-4f4f-a938-83f807904b03-serviceca\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.233592    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.233672    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:14.733650511 +0000 UTC m=+3.051089401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:14.233948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-sysctl-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.233967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-modprobe-d\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-systemd\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-dbus\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.234051    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-host\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-sys\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.234102    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:14.734084841 +0000 UTC m=+3.051523721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-var-lib-kubelet\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82531788-de8e-41d8-92ca-ec35a6e2521d-host-slash\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa356ac2-7704-422c-adac-5885e9238409-kubelet-config\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-lib-modules\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-cnibin\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ef737e1-0408-47c9-962c-ac8ad414f763-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.234616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bcaf0bd-eabc-4f4f-a938-83f807904b03-host\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.235297 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.234671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ef737e1-0408-47c9-962c-ac8ad414f763-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.235297 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.235037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82531788-de8e-41d8-92ca-ec35a6e2521d-iptables-alerter-script\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.235453 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.235299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9edf578f-d23b-45c5-81b6-582b69a24d5f-konnectivity-ca\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.236084 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.236046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9edf578f-d23b-45c5-81b6-582b69a24d5f-agent-certs\") pod \"konnectivity-agent-rcqhx\" (UID: \"9edf578f-d23b-45c5-81b6-582b69a24d5f\") " pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.236452 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.236413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-etc-tuned\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.236732 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.236714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-tmp\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.244164 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.244105    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:14.244164 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.244137    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:14.244164 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.244154    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:14.244394 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.244223    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:14.744204703 +0000 UTC m=+3.061643589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:14.246539 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.246492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtxt\" (UniqueName: \"kubernetes.io/projected/13a33bf6-6706-40de-bdb3-a635e075db13-kube-api-access-5jtxt\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.246635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.246556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9w44\" (UniqueName: \"kubernetes.io/projected/ef28b356-bfe7-4381-89b8-75b9c6a12c5b-kube-api-access-h9w44\") pod \"tuned-xzrm9\" (UID: \"ef28b356-bfe7-4381-89b8-75b9c6a12c5b\") " pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.246836 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.246811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjrf\" (UniqueName: \"kubernetes.io/projected/7bcaf0bd-eabc-4f4f-a938-83f807904b03-kube-api-access-7pjrf\") pod \"node-ca-knslj\" (UID: \"7bcaf0bd-eabc-4f4f-a938-83f807904b03\") " pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.247073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.247050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whr9\" (UniqueName: \"kubernetes.io/projected/82531788-de8e-41d8-92ca-ec35a6e2521d-kube-api-access-5whr9\") pod \"iptables-alerter-vgb77\" (UID: \"82531788-de8e-41d8-92ca-ec35a6e2521d\") " pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.247135 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.247076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdll\" (UniqueName: \"kubernetes.io/projected/5ef737e1-0408-47c9-962c-ac8ad414f763-kube-api-access-smdll\") pod \"multus-additional-cni-plugins-4qw6b\" (UID: \"5ef737e1-0408-47c9-962c-ac8ad414f763\") " pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.322962 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.322928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:14.332401 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.332379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmvp4"
Jul 28 17:17:14.345095 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.345018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m"
Jul 28 17:17:14.349735 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.349712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2m8g7"
Jul 28 17:17:14.357416 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.357388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4qw6b"
Jul 28 17:17:14.365211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.365183    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:14.373143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.373121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzrm9"
Jul 28 17:17:14.381412 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.381383    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgb77"
Jul 28 17:17:14.393284 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.393255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knslj"
Jul 28 17:17:14.735859 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.735822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:14.736039 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.735894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:14.736039 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.736027    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:14.736144 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.736104    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.736083863 +0000 UTC m=+4.053522745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:14.736144 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.736026    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:14.738736 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.736457    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.736332646 +0000 UTC m=+4.053771734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:14.836351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:14.836281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:14.836517 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.836480    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:14.836517 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.836505    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:14.836517 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.836517    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:14.836665 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:14.836587    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.836570355 +0000 UTC m=+4.154009237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:15.015931 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.015905    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef737e1_0408_47c9_962c_ac8ad414f763.slice/crio-b8f5d316acddfcb70e824a73a9e3fdecfa24ab344a277261c8892258e7010a65 WatchSource:0}: Error finding container b8f5d316acddfcb70e824a73a9e3fdecfa24ab344a277261c8892258e7010a65: Status 404 returned error can't find the container with id b8f5d316acddfcb70e824a73a9e3fdecfa24ab344a277261c8892258e7010a65
Jul 28 17:17:15.017172 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.017146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf578f_d23b_45c5_81b6_582b69a24d5f.slice/crio-ae7450fe299e056d51addf667c36d3a71446198d2ce621b6e519e71db20e1b52 WatchSource:0}: Error finding container ae7450fe299e056d51addf667c36d3a71446198d2ce621b6e519e71db20e1b52: Status 404 returned error can't find the container with id ae7450fe299e056d51addf667c36d3a71446198d2ce621b6e519e71db20e1b52
Jul 28 17:17:15.018078 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.018048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1989af2e_a938_451b_a1bf_f5f91e5a07cc.slice/crio-f44e9c4bb883c6a8b8b538ca87355ccf9bc6bd65644fe28d7fe5fd066c3ef88f WatchSource:0}: Error finding container f44e9c4bb883c6a8b8b538ca87355ccf9bc6bd65644fe28d7fe5fd066c3ef88f: Status 404 returned error can't find the container with id f44e9c4bb883c6a8b8b538ca87355ccf9bc6bd65644fe28d7fe5fd066c3ef88f
Jul 28 17:17:15.021008 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.020987    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82531788_de8e_41d8_92ca_ec35a6e2521d.slice/crio-2f14a11679db79676b003647646c383a115a70d7bad884b126bee06808ef0f54 WatchSource:0}: Error finding container 2f14a11679db79676b003647646c383a115a70d7bad884b126bee06808ef0f54: Status 404 returned error can't find the container with id 2f14a11679db79676b003647646c383a115a70d7bad884b126bee06808ef0f54
Jul 28 17:17:15.022805 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.022783    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcaf0bd_eabc_4f4f_a938_83f807904b03.slice/crio-52c8c1dc30fc4c868637f3d3f8a754be1876714d4a42d8ba4925fcb6668c0384 WatchSource:0}: Error finding container 52c8c1dc30fc4c868637f3d3f8a754be1876714d4a42d8ba4925fcb6668c0384: Status 404 returned error can't find the container with id 52c8c1dc30fc4c868637f3d3f8a754be1876714d4a42d8ba4925fcb6668c0384
Jul 28 17:17:15.023681 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.023636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cadb470_ea44_4d68_ba72_673593f2f908.slice/crio-2164d2153e34a19f22a2120d42fe736144cb25577dfa86f1518650a10a003e3e WatchSource:0}: Error finding container 2164d2153e34a19f22a2120d42fe736144cb25577dfa86f1518650a10a003e3e: Status 404 returned error can't find the container with id 2164d2153e34a19f22a2120d42fe736144cb25577dfa86f1518650a10a003e3e
Jul 28 17:17:15.024701 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.024611    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb1f853_e9c4_4d6a_8540_f76de0162149.slice/crio-4da08ebb56ea411b28f64f2e8eef8fbf16ef68646d6218c4de33ebba23b2de05 WatchSource:0}: Error finding container 4da08ebb56ea411b28f64f2e8eef8fbf16ef68646d6218c4de33ebba23b2de05: Status 404 returned error can't find the container with id 4da08ebb56ea411b28f64f2e8eef8fbf16ef68646d6218c4de33ebba23b2de05
Jul 28 17:17:15.026562 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.025896    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef28b356_bfe7_4381_89b8_75b9c6a12c5b.slice/crio-1c131f7083f839cc36c9197cedaa1f826defa6ba06d3b5ccc4f38fff3a44f3c3 WatchSource:0}: Error finding container 1c131f7083f839cc36c9197cedaa1f826defa6ba06d3b5ccc4f38fff3a44f3c3: Status 404 returned error can't find the container with id 1c131f7083f839cc36c9197cedaa1f826defa6ba06d3b5ccc4f38fff3a44f3c3
Jul 28 17:17:15.026562 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:17:15.026493    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5befc373_0bba_4238_839c_7463d45b4913.slice/crio-9f23d0dac74b37c22acf263326fc2b72abad07432e9bc4c011564ea4ce7c7beb WatchSource:0}: Error finding container 9f23d0dac74b37c22acf263326fc2b72abad07432e9bc4c011564ea4ce7c7beb: Status 404 returned error can't find the container with id 9f23d0dac74b37c22acf263326fc2b72abad07432e9bc4c011564ea4ce7c7beb
Jul 28 17:17:15.073239 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.073040    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:14 +0000 UTC" deadline="2027-12-25 06:52:02.050528164 +0000 UTC"
Jul 28 17:17:15.073369 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.073241    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12349h34m46.977292458s"
Jul 28 17:17:15.198901 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.198879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:15.199017 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.198976    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:15.199059 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.199039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:15.199148 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.199132    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:15.208968 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.208943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal" event={"ID":"e04dce7c55a72020d01171561fa4ac33","Type":"ContainerStarted","Data":"d3b8f538a5393b978afd8cda423a7b694921c39d120db61ced7da48299f8b025"}
Jul 28 17:17:15.210044 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.210021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2m8g7" event={"ID":"6cadb470-ea44-4d68-ba72-673593f2f908","Type":"ContainerStarted","Data":"2164d2153e34a19f22a2120d42fe736144cb25577dfa86f1518650a10a003e3e"}
Jul 28 17:17:15.210950 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.210927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgb77" event={"ID":"82531788-de8e-41d8-92ca-ec35a6e2521d","Type":"ContainerStarted","Data":"2f14a11679db79676b003647646c383a115a70d7bad884b126bee06808ef0f54"}
Jul 28 17:17:15.211920 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.211902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m" event={"ID":"1989af2e-a938-451b-a1bf-f5f91e5a07cc","Type":"ContainerStarted","Data":"f44e9c4bb883c6a8b8b538ca87355ccf9bc6bd65644fe28d7fe5fd066c3ef88f"}
Jul 28 17:17:15.212950 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.212921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcqhx" event={"ID":"9edf578f-d23b-45c5-81b6-582b69a24d5f","Type":"ContainerStarted","Data":"ae7450fe299e056d51addf667c36d3a71446198d2ce621b6e519e71db20e1b52"}
Jul 28 17:17:15.213949 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.213929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerStarted","Data":"b8f5d316acddfcb70e824a73a9e3fdecfa24ab344a277261c8892258e7010a65"}
Jul 28 17:17:15.214958 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.214930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"9f23d0dac74b37c22acf263326fc2b72abad07432e9bc4c011564ea4ce7c7beb"}
Jul 28 17:17:15.215886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.215865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzrm9" event={"ID":"ef28b356-bfe7-4381-89b8-75b9c6a12c5b","Type":"ContainerStarted","Data":"1c131f7083f839cc36c9197cedaa1f826defa6ba06d3b5ccc4f38fff3a44f3c3"}
Jul 28 17:17:15.216765 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.216740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmvp4" event={"ID":"ecb1f853-e9c4-4d6a-8540-f76de0162149","Type":"ContainerStarted","Data":"4da08ebb56ea411b28f64f2e8eef8fbf16ef68646d6218c4de33ebba23b2de05"}
Jul 28 17:17:15.217632 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.217616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knslj" event={"ID":"7bcaf0bd-eabc-4f4f-a938-83f807904b03","Type":"ContainerStarted","Data":"52c8c1dc30fc4c868637f3d3f8a754be1876714d4a42d8ba4925fcb6668c0384"}
Jul 28 17:17:15.225126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.225082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-92.ec2.internal" podStartSLOduration=2.22506872 podStartE2EDuration="2.22506872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:15.224917933 +0000 UTC m=+3.542356813" watchObservedRunningTime="2026-07-28 17:17:15.22506872 +0000 UTC m=+3.542507607"
Jul 28 17:17:15.744828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.744785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:15.745009 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.744862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:15.745009 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.744964    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.745009 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.744978    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:15.745175 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.745035    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:17.74501695 +0000 UTC m=+6.062455818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:15.745175 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.745057    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:17.745047256 +0000 UTC m=+6.062486126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.846742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:15.846088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:15.846742 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.846275    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:15.846742 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.846317    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:15.846742 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.846331    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:15.846742 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:15.846389    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:17.846370526 +0000 UTC m=+6.163809394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:16.074493 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:16.074399    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:14 +0000 UTC" deadline="2028-01-12 18:38:17.98703232 +0000 UTC"
Jul 28 17:17:16.074493 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:16.074441    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12793h21m1.912595747s"
Jul 28 17:17:16.200279 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:16.200249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:16.200469 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:16.200399    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:16.232708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:16.231763    2561 generic.go:356] "Generic (PLEG): container finished" podID="1558a3aa7615c15900b8d30d171fa55d" containerID="58e8d842b2e3dcd050a46581cdedbc2ec8d36cb4acd8c53999a17ae132ec0d06" exitCode=0
Jul 28 17:17:16.232708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:16.232564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal" event={"ID":"1558a3aa7615c15900b8d30d171fa55d","Type":"ContainerDied","Data":"58e8d842b2e3dcd050a46581cdedbc2ec8d36cb4acd8c53999a17ae132ec0d06"}
Jul 28 17:17:17.198600 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.198562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:17.199097 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.198694    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:17.199097 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.199075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:17.199200 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.199163    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:17.247526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.247080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal" event={"ID":"1558a3aa7615c15900b8d30d171fa55d","Type":"ContainerStarted","Data":"33b536689d61be7b432721e46a440b3b8d0b12042e9aa403ce4f372c1f2bb86c"}
Jul 28 17:17:17.760819 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.760766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:17.761011 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.760840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:17.761011 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.760980    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.761117 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.761043    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.761025297 +0000 UTC m=+10.078464164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.761577 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.761446    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:17.761577 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.761498    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.761482798 +0000 UTC m=+10.078921665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:17.862050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:17.862014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:17.862335 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.862297    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:17.862439 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.862339    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:17.862439 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.862353    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:17.862439 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:17.862436    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.862415234 +0000 UTC m=+10.179854118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:18.199805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:18.199423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:18.199805 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:18.199540    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:19.198335 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:19.198281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:19.198532 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:19.198343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:19.198532 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:19.198432    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:19.198685 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:19.198557    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:20.198973 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:20.198940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:20.199438 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:20.199090    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:21.198577 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:21.198112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:21.198577 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.198257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:21.198577 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:21.198333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:21.198577 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.198490    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:21.795162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:21.795128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:21.795557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:21.795181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:21.795557 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.795295    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:21.795557 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.795297    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:21.795557 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.795369    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:29.795351738 +0000 UTC m=+18.112790604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:21.795557 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.795384    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:29.795377445 +0000 UTC m=+18.112816309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:21.896067 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:21.896032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:21.896241 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.896221    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:21.896288 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.896249    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:21.896288 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.896263    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:21.896402 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:21.896337    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:29.896300554 +0000 UTC m=+18.213739444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:22.198678 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:22.198650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:22.198856 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:22.198788    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:23.198676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:23.198641    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:23.199121 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:23.198772    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:23.199121 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:23.198873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:23.199121 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:23.198981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:24.199063 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:24.199031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:24.199517 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:24.199176    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:25.198823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:25.198787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:25.199011 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:25.198787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:25.199011 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:25.198907    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:25.199099 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:25.199019    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:26.198478 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:26.198401    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:26.198622 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:26.198557    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:27.198706 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:27.198676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:27.199153 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:27.198781    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:27.199153 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:27.198851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:27.199153 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:27.198956    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:28.198822 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:28.198791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:28.199267 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:28.198938    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:29.198342 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:29.198244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:29.198342 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:29.198297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:29.198656 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.198402    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:29.198656 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.198543    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:29.854062 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:29.854020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:29.854584 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:29.854107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:29.854584 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.854186    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:29.854584 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.854234    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:29.854584 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.854261    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.854244565 +0000 UTC m=+34.171683446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:29.854584 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.854284    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.854268051 +0000 UTC m=+34.171706917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:29.954526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:29.954487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:29.954709 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.954626    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:29.954709 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.954651    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:29.954709 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.954662    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:29.954874 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:29.954713    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.954698626 +0000 UTC m=+34.272137491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:30.198447 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:30.198410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:30.198627 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:30.198546    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:31.198722 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:31.198687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:31.199132 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:31.198697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:31.199132 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:31.198801    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:31.199132 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:31.198884    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:32.198671 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:32.198636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:32.198842 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:32.198729    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:33.198252 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:33.198196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:33.198452 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:33.198216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:33.198452 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:33.198343    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:33.198452 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:33.198416    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:34.198270 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.197955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:34.198830 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:34.198327    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:34.285923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.285690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2m8g7" event={"ID":"6cadb470-ea44-4d68-ba72-673593f2f908","Type":"ContainerStarted","Data":"5050468f091114f7ebb4c985824304e453d5c5f7aeb3a23751638077c773d5b9"}
Jul 28 17:17:34.287141 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.287107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m" event={"ID":"1989af2e-a938-451b-a1bf-f5f91e5a07cc","Type":"ContainerStarted","Data":"2e63d4552e2c51b397bd4117e398a6dda007df35263f98efbd4097db545cf959"}
Jul 28 17:17:34.288455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.288417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcqhx" event={"ID":"9edf578f-d23b-45c5-81b6-582b69a24d5f","Type":"ContainerStarted","Data":"4081f2cb4663800ff3d7cca01e8e02d907b8672feafaa765cebaddc0db06288c"}
Jul 28 17:17:34.290252 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.290231    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="a7d3a08826ce53b1ad02f614b1004ada8b20e446115d922c1e3c6c9f4e0ae7da" exitCode=0
Jul 28 17:17:34.290350 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.290262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"a7d3a08826ce53b1ad02f614b1004ada8b20e446115d922c1e3c6c9f4e0ae7da"}
Jul 28 17:17:34.292221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.292101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"85ae87f3199b4457edf61cf63b7f76a5f8d63a88eab5587c36f2591b36245d1a"}
Jul 28 17:17:34.293963 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.293927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzrm9" event={"ID":"ef28b356-bfe7-4381-89b8-75b9c6a12c5b","Type":"ContainerStarted","Data":"8ca41b4b0546b54f25ed6c40e12b28a3398d22fbfece11b53b90d5b54686dc07"}
Jul 28 17:17:34.295508 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.295485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmvp4" event={"ID":"ecb1f853-e9c4-4d6a-8540-f76de0162149","Type":"ContainerStarted","Data":"22c051dd4ee72ec5a8091bdd76c37e4e8acf742d0ca2df7c6fe92f0f95e95ab0"}
Jul 28 17:17:34.297044 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.297024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knslj" event={"ID":"7bcaf0bd-eabc-4f4f-a938-83f807904b03","Type":"ContainerStarted","Data":"ae3fc68553b1f9ab2f10a03968fe41104ffff7cb952bc4ec30666889065278e7"}
Jul 28 17:17:34.308481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.308427    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-92.ec2.internal" podStartSLOduration=21.308402558 podStartE2EDuration="21.308402558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:17.276748498 +0000 UTC m=+5.594187387" watchObservedRunningTime="2026-07-28 17:17:34.308402558 +0000 UTC m=+22.625841445"
Jul 28 17:17:34.330934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.330891    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2m8g7" podStartSLOduration=3.378800304 podStartE2EDuration="22.330876623s" totalImagesPullingTime="18.952076319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.307597798 +0000 UTC m=+22.625036686" watchObservedRunningTime="2026-07-28 17:17:34.330876623 +0000 UTC m=+22.648315510"
Jul 28 17:17:34.349458 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.349405    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rcqhx" podStartSLOduration=3.426574319 podStartE2EDuration="22.349387013s" totalImagesPullingTime="18.922812694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.330605288 +0000 UTC m=+22.648044175" watchObservedRunningTime="2026-07-28 17:17:34.349387013 +0000 UTC m=+22.666825903"
Jul 28 17:17:34.362926 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.362874    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xzrm9" podStartSLOduration=3.4486042 podStartE2EDuration="22.362857963s" totalImagesPullingTime="18.914253763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.348762758 +0000 UTC m=+22.666201645" watchObservedRunningTime="2026-07-28 17:17:34.362857963 +0000 UTC m=+22.680296904"
Jul 28 17:17:34.363121 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.363051    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knslj" podStartSLOduration=3.427056917 podStartE2EDuration="22.363042974s" totalImagesPullingTime="18.935986057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.362993162 +0000 UTC m=+22.680432032" watchObservedRunningTime="2026-07-28 17:17:34.363042974 +0000 UTC m=+22.680481862"
Jul 28 17:17:34.377588 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:34.377416    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dmvp4" podStartSLOduration=3.442292709 podStartE2EDuration="22.377391863s" totalImagesPullingTime="18.935099154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.377293325 +0000 UTC m=+22.694732212" watchObservedRunningTime="2026-07-28 17:17:34.377391863 +0000 UTC m=+22.694830747"
Jul 28 17:17:35.106977 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.106951    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knslj_7bcaf0bd-eabc-4f4f-a938-83f807904b03/node-ca/0.log"
Jul 28 17:17:35.198275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.198242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:35.198886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.198242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:35.198886 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:35.198380    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:35.198886 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:35.198415    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:35.301242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.301209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"44cb11614963e682b1867dd9121ef9f159a8fae42b63570f5fb48f843e7c5bf8"}
Jul 28 17:17:35.301462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.301295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"1ccfc94cafa49bba641d64b54492d2078f80c1e99a3d04140497a2fa6c14733e"}
Jul 28 17:17:35.301462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.301360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"379f11e2cf7081d063c25bd37126427168081a5ec963143737b53ca0d69c367b"}
Jul 28 17:17:35.301462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.301406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"79a012f1cab98ff43228a52b94fec03d8c582e7fad6a9bcdb36c2e6ef62042a0"}
Jul 28 17:17:35.301462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.301453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"93cf551de7689f46901fefbc17cca42d6545da8a047fa20d701bfa23e366db72"}
Jul 28 17:17:35.550662 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:35.550625    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:36.097480 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.097361    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:35.55064764Z","UUID":"b61c5425-301d-42c1-b9fc-bdaaf31aa70c","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:36.098955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.098934    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:36.099092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.098975    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:36.199202 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.198567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:36.199202 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:36.198731    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:36.305197 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.305155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgb77" event={"ID":"82531788-de8e-41d8-92ca-ec35a6e2521d","Type":"ContainerStarted","Data":"67dd03156ef94f102332aa310c7de4b3b1e1f57e106137c5e900fe56367cd795"}
Jul 28 17:17:36.307796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.307758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m" event={"ID":"1989af2e-a938-451b-a1bf-f5f91e5a07cc","Type":"ContainerStarted","Data":"0f46256636d02410164f68175159e1e3b3f9de40dbe01265b3329748c099d921"}
Jul 28 17:17:36.344581 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:36.344528    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vgb77" podStartSLOduration=5.425011985 podStartE2EDuration="24.3445122s" totalImagesPullingTime="18.919500215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:36.344289782 +0000 UTC m=+24.661728669" watchObservedRunningTime="2026-07-28 17:17:36.3445122 +0000 UTC m=+24.661951448"
Jul 28 17:17:37.198967 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:37.198932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:37.199161 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:37.198932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:37.199161 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:37.199107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:37.199161 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:37.199149    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:37.313784 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:37.313747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"17c0062c9defa12b4195f89e93b89960fa723ab04d7333f11e1eef0462fca204"}
Jul 28 17:17:37.316148 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:37.316118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m" event={"ID":"1989af2e-a938-451b-a1bf-f5f91e5a07cc","Type":"ContainerStarted","Data":"23ec36c4a2735b181cc9945a7b023dcf959aef5e9d9f7c058771c40d45cbaebf"}
Jul 28 17:17:37.336540 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:37.336486    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnc4m" podStartSLOduration=4.095004585 podStartE2EDuration="25.336467787s" totalImagesPullingTime="21.241463202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:37.335834129 +0000 UTC m=+25.653273043" watchObservedRunningTime="2026-07-28 17:17:37.336467787 +0000 UTC m=+25.653906674"
Jul 28 17:17:38.199240 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:38.199050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:38.199707 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:38.199361    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:38.354663 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:38.354629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:38.355315 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:38.355289    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:39.199214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.198966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:39.199214 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:39.199066    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:39.200127 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.199419    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:39.200127 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:39.199506    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:39.323813 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.323562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerStarted","Data":"078b2445d858966d11c58d1cbc52195789bf58b7f7aa1d0a184c00ac4ebb0171"}
Jul 28 17:17:39.327087 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.327057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" event={"ID":"5befc373-0bba-4238-839c-7463d45b4913","Type":"ContainerStarted","Data":"fa27d25fc57c16fccea76f30db737a13507156f0bffa0917a31b80c9cfcda466"}
Jul 28 17:17:39.327426 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.327405    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:39.327546 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.327445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:39.327771 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.327757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rcqhx"
Jul 28 17:17:39.343624 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.343591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:39.371641 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:39.371590    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5" podStartSLOduration=8.268645003 podStartE2EDuration="27.371571822s" totalImagesPullingTime="19.102926819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:39.371026084 +0000 UTC m=+27.688464971" watchObservedRunningTime="2026-07-28 17:17:39.371571822 +0000 UTC m=+27.689010708"
Jul 28 17:17:40.198939 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.198902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:40.199076 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:40.199013    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:40.330794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.330760    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="078b2445d858966d11c58d1cbc52195789bf58b7f7aa1d0a184c00ac4ebb0171" exitCode=0
Jul 28 17:17:40.331236 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.330838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"078b2445d858966d11c58d1cbc52195789bf58b7f7aa1d0a184c00ac4ebb0171"}
Jul 28 17:17:40.331236 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.330902    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:17:40.331421 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.331397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:40.347648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:40.347614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:41.198526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:41.198487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:41.198526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:41.198514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:41.198737 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:41.198618    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:41.198737 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:41.198716    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:41.332688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:41.332664    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:17:42.035698 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:42.035667    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:17:42.199482 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:42.199442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:42.199664 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:42.199576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:42.336777 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:42.336688    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="d809d3aaebcdba96431592d372d61cb1de3848dee7f4dd5645370e268fe7a172" exitCode=0
Jul 28 17:17:42.337188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:42.336769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"d809d3aaebcdba96431592d372d61cb1de3848dee7f4dd5645370e268fe7a172"}
Jul 28 17:17:43.198799 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:43.198756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:43.198986 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:43.198872    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:43.198986 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:43.198933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:43.199059 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:43.199029    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:44.198828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:44.198794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:44.199206 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:44.198909    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:44.344043 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:44.344005    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="412de36774bad98a40adc565cfdec568102fe836f8c136f0d1e9592e0eab4b09" exitCode=0
Jul 28 17:17:44.344207 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:44.344066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"412de36774bad98a40adc565cfdec568102fe836f8c136f0d1e9592e0eab4b09"}
Jul 28 17:17:45.198218 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:45.198176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:45.198414 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:45.198182    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:45.198414 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.198317    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:45.198542 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.198417    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:45.873443 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:45.873407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:45.873902 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:45.873458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:45.873902 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.873569    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:45.873902 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.873633    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret podName:fa356ac2-7704-422c-adac-5885e9238409 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:17.873613878 +0000 UTC m=+66.191052757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret") pod "global-pull-secret-syncer-rw72f" (UID: "fa356ac2-7704-422c-adac-5885e9238409") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:45.873902 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.873570    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:45.873902 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.873731    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs podName:13a33bf6-6706-40de-bdb3-a635e075db13 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:17.873707331 +0000 UTC m=+66.191146214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs") pod "network-metrics-daemon-7nphk" (UID: "13a33bf6-6706-40de-bdb3-a635e075db13") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:45.974706 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:45.974667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:45.974908 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.974859    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:45.974908 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.974887    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:45.974908 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.974896    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qn7kz for pod openshift-network-diagnostics/network-check-target-sck52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:45.975047 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:45.974953    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz podName:8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:17.974936246 +0000 UTC m=+66.292375128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn7kz" (UniqueName: "kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz") pod "network-check-target-sck52" (UID: "8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:46.199026 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:46.198989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:46.199223 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:46.199138    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:47.198104 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:47.198076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:47.198592 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:47.198190    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:47.198592 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:47.198248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:47.198592 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:47.198366    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:48.198555 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:48.198529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:48.198957 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:48.198661    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:48.356000 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:48.355963    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="06b45895794bd0149e5b6af210ce343f72a0b335c6c525802c4769b0c16500e2" exitCode=0
Jul 28 17:17:48.356156 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:48.356026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"06b45895794bd0149e5b6af210ce343f72a0b335c6c525802c4769b0c16500e2"}
Jul 28 17:17:49.198378 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:49.198340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:49.198534 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:49.198340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:49.198534 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:49.198451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:49.198534 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:49.198512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:49.361049 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:49.361014    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ef737e1-0408-47c9-962c-ac8ad414f763" containerID="2227d6a98f9344e580535b5949e6a6a093cc93c4c494defa43a6dc319ab98f54" exitCode=0
Jul 28 17:17:49.361193 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:49.361058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerDied","Data":"2227d6a98f9344e580535b5949e6a6a093cc93c4c494defa43a6dc319ab98f54"}
Jul 28 17:17:50.198511 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:50.198334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:50.198711 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:50.198588    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:50.366020 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:50.365989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" event={"ID":"5ef737e1-0408-47c9-962c-ac8ad414f763","Type":"ContainerStarted","Data":"23e4b904ba925bfb05a16ccf979784dcde5747d929f903d8dba680d6a0fa1823"}
Jul 28 17:17:50.396767 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:50.396721    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4qw6b" podStartSLOduration=8.749567845 podStartE2EDuration="38.396703325s" totalImagesPullingTime="29.64713548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:50.396273879 +0000 UTC m=+38.713712764" watchObservedRunningTime="2026-07-28 17:17:50.396703325 +0000 UTC m=+38.714142213"
Jul 28 17:17:51.198785 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:51.198752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:51.199180 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:51.198846    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:51.199180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:51.198872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:51.199180 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:51.198920    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:52.199201 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:52.199164    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:52.199645 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:52.199271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:53.198787 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:53.198752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:53.198998 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:53.198867    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:53.198998 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:53.198912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:53.198998 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:53.198991    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:54.201263 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:54.201233    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:54.201642 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:54.201356    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:55.198519 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:55.198479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:55.198690 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:55.198479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:55.198690 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:55.198595    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:55.198690 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:55.198651    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:56.198794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:56.198759    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:56.199154 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:56.198865    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:57.198619 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:57.198590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:57.198787 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:57.198590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:57.198787 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:57.198698    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:57.198787 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:57.198752    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:58.198698 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:58.198665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:58.198863 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:58.198800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:59.198795 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.198762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:59.198927 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:59.198861    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:59.198927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.198912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:59.199620 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:59.198996    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:17:59.361171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.361141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw72f"]
Jul 28 17:17:59.365479 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.365442    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sck52"]
Jul 28 17:17:59.366091 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.366060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7nphk"]
Jul 28 17:17:59.366243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.366152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:17:59.366335 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:59.366264    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:17:59.385808 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.385746    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:17:59.385808 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:17:59.385766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:17:59.385959 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:59.385837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:17:59.386009 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:17:59.385980    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:18:01.198466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:01.198433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:01.198976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:01.198544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:18:01.198976 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:01.198556    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sck52" podUID="8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92"
Jul 28 17:18:01.198976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:01.198573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:18:01.198976 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:01.198644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw72f" podUID="fa356ac2-7704-422c-adac-5885e9238409"
Jul 28 17:18:01.198976 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:01.198710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nphk" podUID="13a33bf6-6706-40de-bdb3-a635e075db13"
Jul 28 17:18:03.035109 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.035082    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-92.ec2.internal" event="NodeReady"
Jul 28 17:18:03.035570 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.035192    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:18:03.099153 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.099120    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mh54r"]
Jul 28 17:18:03.114195 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.114154    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gdrjx"]
Jul 28 17:18:03.114382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.114279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.119356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.119287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-842mx\""
Jul 28 17:18:03.119864 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.119848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:18:03.119990 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.119972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:18:03.128573 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.128380    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mh54r"]
Jul 28 17:18:03.128708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.128593    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdrjx"]
Jul 28 17:18:03.128708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.128502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.131455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.131391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:18:03.131614 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.131597    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:18:03.131689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.131656    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:18:03.131689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.131667    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:18:03.131816 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.131796    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdvg7\""
Jul 28 17:18:03.133538 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.133511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7vqvq"]
Jul 28 17:18:03.158366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.158334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vqvq"]
Jul 28 17:18:03.158523 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.158429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.161180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.161159    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:18:03.161327 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.161197    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:18:03.161327 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.161197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rk7tv\""
Jul 28 17:18:03.161565 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.161552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:18:03.167698 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.167677    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:03.186442 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.186408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:03.186587 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.186511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.189685 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.189661    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:18:03.190035 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:18:03.190035 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:18:03.190190 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190125    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:18:03.190284 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190268    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sn7jd\""
Jul 28 17:18:03.190359 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190334    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:18:03.190483 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:18:03.190980 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.190963    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:18:03.198767 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.198747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:18:03.198873 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.198841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:03.199064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.199048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:18:03.201648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.201632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:18:03.201720 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.201689    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:03.202360 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.202344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmkkk\""
Jul 28 17:18:03.202410 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.202398    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:18:03.202454 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.202439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lx2z\""
Jul 28 17:18:03.202710 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.202699    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:03.207598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-tmp-dir\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.207705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/289b8896-51ca-4643-8767-f34ba8c2ab88-data-volume\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.207705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.207705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.207849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.207849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-kube-api-access-fjvx8\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.207849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgp58\" (UniqueName: \"kubernetes.io/projected/be2708eb-224f-4dd7-bfa0-0fc62342849d-kube-api-access-zgp58\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.207990 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2vs\" (UniqueName: \"kubernetes.io/projected/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-api-access-pw2vs\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.207990 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.207981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/289b8896-51ca-4643-8767-f34ba8c2ab88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.208087 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xsz\" (UniqueName: \"kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.208087 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-metrics-tls\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.208087 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.208230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.208230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.208230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2708eb-224f-4dd7-bfa0-0fc62342849d-cert\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.208230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-config-volume\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.208441 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.208236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/289b8896-51ca-4643-8767-f34ba8c2ab88-crio-socket\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.308818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-config-volume\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.308818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/289b8896-51ca-4643-8767-f34ba8c2ab88-crio-socket\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.308818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-tmp-dir\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/289b8896-51ca-4643-8767-f34ba8c2ab88-data-volume\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-kube-api-access-fjvx8\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgp58\" (UniqueName: \"kubernetes.io/projected/be2708eb-224f-4dd7-bfa0-0fc62342849d-kube-api-access-zgp58\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.308957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2vs\" (UniqueName: \"kubernetes.io/projected/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-api-access-pw2vs\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/289b8896-51ca-4643-8767-f34ba8c2ab88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/289b8896-51ca-4643-8767-f34ba8c2ab88-crio-socket\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xsz\" (UniqueName: \"kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-metrics-tls\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.309100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/289b8896-51ca-4643-8767-f34ba8c2ab88-data-volume\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2708eb-224f-4dd7-bfa0-0fc62342849d-cert\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-config-volume\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.309712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.309177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-tmp-dir\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.310341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.310242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.310341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.310323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.310729 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.310711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.310894 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.310874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.313383 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.313357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-metrics-tls\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.313596 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.313442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.313596 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.313475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/289b8896-51ca-4643-8767-f34ba8c2ab88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.313596 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.313520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be2708eb-224f-4dd7-bfa0-0fc62342849d-cert\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.313596 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.313539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.318208 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.318180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/e281de0b-41cf-4e5d-bb3f-263bd543b1f1-kube-api-access-fjvx8\") pod \"dns-default-mh54r\" (UID: \"e281de0b-41cf-4e5d-bb3f-263bd543b1f1\") " pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.318722 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.318705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2vs\" (UniqueName: \"kubernetes.io/projected/289b8896-51ca-4643-8767-f34ba8c2ab88-kube-api-access-pw2vs\") pod \"insights-runtime-extractor-gdrjx\" (UID: \"289b8896-51ca-4643-8767-f34ba8c2ab88\") " pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.318973 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.318952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgp58\" (UniqueName: \"kubernetes.io/projected/be2708eb-224f-4dd7-bfa0-0fc62342849d-kube-api-access-zgp58\") pod \"ingress-canary-7vqvq\" (UID: \"be2708eb-224f-4dd7-bfa0-0fc62342849d\") " pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.319248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.319233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xsz\" (UniqueName: \"kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz\") pod \"console-584c5d6559-8w77b\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") " pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.423851 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.423815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:03.439007 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.438980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gdrjx"
Jul 28 17:18:03.466755 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.466723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vqvq"
Jul 28 17:18:03.495927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.495702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:03.655323 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.655277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gdrjx"]
Jul 28 17:18:03.658467 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:03.658437    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289b8896_51ca_4643_8767_f34ba8c2ab88.slice/crio-ca9226507ba180912f77a112d7822a720c5d359e834df2ef23ee64eaa215a155 WatchSource:0}: Error finding container ca9226507ba180912f77a112d7822a720c5d359e834df2ef23ee64eaa215a155: Status 404 returned error can't find the container with id ca9226507ba180912f77a112d7822a720c5d359e834df2ef23ee64eaa215a155
Jul 28 17:18:03.662723 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.662697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mh54r"]
Jul 28 17:18:03.664222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.664160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vqvq"]
Jul 28 17:18:03.667372 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:03.667348    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281de0b_41cf_4e5d_bb3f_263bd543b1f1.slice/crio-b346b5df67c7c4317d0ae4b75bc8bbf091d1b38f3cda5cec42fc4fa36de37bc8 WatchSource:0}: Error finding container b346b5df67c7c4317d0ae4b75bc8bbf091d1b38f3cda5cec42fc4fa36de37bc8: Status 404 returned error can't find the container with id b346b5df67c7c4317d0ae4b75bc8bbf091d1b38f3cda5cec42fc4fa36de37bc8
Jul 28 17:18:03.668098 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:03.668031    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2708eb_224f_4dd7_bfa0_0fc62342849d.slice/crio-07ad6de2aa2e4a2411b1fae3499af208742d0be6421cbca12d45dba1b9247bee WatchSource:0}: Error finding container 07ad6de2aa2e4a2411b1fae3499af208742d0be6421cbca12d45dba1b9247bee: Status 404 returned error can't find the container with id 07ad6de2aa2e4a2411b1fae3499af208742d0be6421cbca12d45dba1b9247bee
Jul 28 17:18:03.685084 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.685061    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:03.688153 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:03.688133    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e73fec9_2e94_44ee_82f0_22ebc8488634.slice/crio-25d6c43363ca6bc918741c5576a2c4f6e48b4cb7241544c61c23cf2aa965cfd3 WatchSource:0}: Error finding container 25d6c43363ca6bc918741c5576a2c4f6e48b4cb7241544c61c23cf2aa965cfd3: Status 404 returned error can't find the container with id 25d6c43363ca6bc918741c5576a2c4f6e48b4cb7241544c61c23cf2aa965cfd3
Jul 28 17:18:03.930493 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.930463    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:18:03.954050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.954018    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:18:03.954210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.954115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:03.968057 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:03.968034    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:18:04.015811 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015811 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfrp\" (UniqueName: \"kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.015975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.015928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.116971 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.116942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfrp\" (UniqueName: \"kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117386 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.117915 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.118013 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.117923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.118091 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.118070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.118145 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.118072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.120299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.120282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.120377 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.120284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.125895 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.125872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfrp\" (UniqueName: \"kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp\") pod \"console-67564fb4b-2s27v\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") " pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.263051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.262973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:04.398136 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.398044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdrjx" event={"ID":"289b8896-51ca-4643-8767-f34ba8c2ab88","Type":"ContainerStarted","Data":"ca9226507ba180912f77a112d7822a720c5d359e834df2ef23ee64eaa215a155"}
Jul 28 17:18:04.399397 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.399285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584c5d6559-8w77b" event={"ID":"5e73fec9-2e94-44ee-82f0-22ebc8488634","Type":"ContainerStarted","Data":"25d6c43363ca6bc918741c5576a2c4f6e48b4cb7241544c61c23cf2aa965cfd3"}
Jul 28 17:18:04.400819 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.400793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vqvq" event={"ID":"be2708eb-224f-4dd7-bfa0-0fc62342849d","Type":"ContainerStarted","Data":"07ad6de2aa2e4a2411b1fae3499af208742d0be6421cbca12d45dba1b9247bee"}
Jul 28 17:18:04.402884 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.402860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh54r" event={"ID":"e281de0b-41cf-4e5d-bb3f-263bd543b1f1","Type":"ContainerStarted","Data":"b346b5df67c7c4317d0ae4b75bc8bbf091d1b38f3cda5cec42fc4fa36de37bc8"}
Jul 28 17:18:04.439433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:04.438872    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:18:04.444250 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:04.444219    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0832de7_95d2_409e_8644_f86e0d49a297.slice/crio-2f7221d0718edb4f6158cbb69ed3dc64e521a70ac9a50862c8d7d48f8f694606 WatchSource:0}: Error finding container 2f7221d0718edb4f6158cbb69ed3dc64e521a70ac9a50862c8d7d48f8f694606: Status 404 returned error can't find the container with id 2f7221d0718edb4f6158cbb69ed3dc64e521a70ac9a50862c8d7d48f8f694606
Jul 28 17:18:05.414013 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:05.413777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67564fb4b-2s27v" event={"ID":"e0832de7-95d2-409e-8644-f86e0d49a297","Type":"ContainerStarted","Data":"2f7221d0718edb4f6158cbb69ed3dc64e521a70ac9a50862c8d7d48f8f694606"}
Jul 28 17:18:12.444232 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.444013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdrjx" event={"ID":"289b8896-51ca-4643-8767-f34ba8c2ab88","Type":"ContainerStarted","Data":"f4eeef6cef3f43b79241dc55e84adfec9bdcdde182dfee6ecc784f3924e92fb9"}
Jul 28 17:18:12.446585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.446547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584c5d6559-8w77b" event={"ID":"5e73fec9-2e94-44ee-82f0-22ebc8488634","Type":"ContainerStarted","Data":"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"}
Jul 28 17:18:12.448622 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.448490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vqvq" event={"ID":"be2708eb-224f-4dd7-bfa0-0fc62342849d","Type":"ContainerStarted","Data":"30d601d827d2d8b04c8bf27cc378acd9192cba20a340765ac07bc4c03e547952"}
Jul 28 17:18:12.450931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.450751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67564fb4b-2s27v" event={"ID":"e0832de7-95d2-409e-8644-f86e0d49a297","Type":"ContainerStarted","Data":"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"}
Jul 28 17:18:12.453188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.453106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh54r" event={"ID":"e281de0b-41cf-4e5d-bb3f-263bd543b1f1","Type":"ContainerStarted","Data":"323935c7c222b8d7b39deff11226b46dde7cdea79c20909219012ac83ad8744c"}
Jul 28 17:18:12.453188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.453153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh54r" event={"ID":"e281de0b-41cf-4e5d-bb3f-263bd543b1f1","Type":"ContainerStarted","Data":"885ad460051789d1cb1ef4a7caaa9e0a992dbd0550297522c4b19442a8dbc1a6"}
Jul 28 17:18:12.453387 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.453240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:12.496109 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.495734    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-584c5d6559-8w77b" podStartSLOduration=0.991876959 podStartE2EDuration="9.49569589s" totalImagesPullingTime="8.503818931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:12.495676722 +0000 UTC m=+60.813115611" watchObservedRunningTime="2026-07-28 17:18:12.49569589 +0000 UTC m=+60.813134776"
Jul 28 17:18:12.522819 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.522769    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67564fb4b-2s27v" podStartSLOduration=1.777478261 podStartE2EDuration="9.522751146s" totalImagesPullingTime="7.745272885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:12.521919112 +0000 UTC m=+60.839357999" watchObservedRunningTime="2026-07-28 17:18:12.522751146 +0000 UTC m=+60.840190022"
Jul 28 17:18:12.576719 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.576664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mh54r" podStartSLOduration=1.433164372 podStartE2EDuration="9.576646071s" totalImagesPullingTime="8.143481699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:12.576489455 +0000 UTC m=+60.893928343" watchObservedRunningTime="2026-07-28 17:18:12.576646071 +0000 UTC m=+60.894085000"
Jul 28 17:18:12.577089 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:12.577067    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7vqvq" podStartSLOduration=1.053734899 podStartE2EDuration="9.577060672s" totalImagesPullingTime="8.523325773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:12.546926418 +0000 UTC m=+60.864365304" watchObservedRunningTime="2026-07-28 17:18:12.577060672 +0000 UTC m=+60.894499537"
Jul 28 17:18:13.353681 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:13.353648    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6spk5"
Jul 28 17:18:13.496819 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:13.496782    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:13.497284 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:13.496881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:13.498676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:13.498646    2561 patch_prober.go:28] interesting pod/console-584c5d6559-8w77b container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Jul 28 17:18:13.498825 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:13.498719    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-584c5d6559-8w77b" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Jul 28 17:18:14.263977 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:14.263936    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:14.264167 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:14.264011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:14.265467 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:14.265439    2561 patch_prober.go:28] interesting pod/console-67564fb4b-2s27v container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body=
Jul 28 17:18:14.265568 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:14.265490    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-67564fb4b-2s27v" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" containerName="console" probeResult="failure" output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused"
Jul 28 17:18:15.266075 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.266037    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"]
Jul 28 17:18:15.283527 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.283479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"]
Jul 28 17:18:15.283715 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.283600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.286492 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.286463    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:18:15.286647 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.286537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:18:15.286647 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.286636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:18:15.286790 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.286636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:15.287016 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.286990    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:18:15.287788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.287768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rncn6\""
Jul 28 17:18:15.320886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.320852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc30941-eb09-4145-8f16-c72d17284331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.320886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.320894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpb2\" (UniqueName: \"kubernetes.io/projected/ecc30941-eb09-4145-8f16-c72d17284331-kube-api-access-5kpb2\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.321105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.320945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.321105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.320972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.422184 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.422145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.422184 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.422189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.422447 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.422265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc30941-eb09-4145-8f16-c72d17284331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.422447 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.422281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpb2\" (UniqueName: \"kubernetes.io/projected/ecc30941-eb09-4145-8f16-c72d17284331-kube-api-access-5kpb2\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.422554 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:15.422486    2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:18:15.422608 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:15.422575    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls podName:ecc30941-eb09-4145-8f16-c72d17284331 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:15.922551188 +0000 UTC m=+64.239990074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-rrmhq" (UID: "ecc30941-eb09-4145-8f16-c72d17284331") : secret "prometheus-operator-tls" not found
Jul 28 17:18:15.422979 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.422956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc30941-eb09-4145-8f16-c72d17284331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.425349 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.425327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.432139 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.432116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpb2\" (UniqueName: \"kubernetes.io/projected/ecc30941-eb09-4145-8f16-c72d17284331-kube-api-access-5kpb2\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.926471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:15.926428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:15.926652 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:15.926595    2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:18:15.926714 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:15.926678    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls podName:ecc30941-eb09-4145-8f16-c72d17284331 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:16.926655174 +0000 UTC m=+65.244094057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-rrmhq" (UID: "ecc30941-eb09-4145-8f16-c72d17284331") : secret "prometheus-operator-tls" not found
Jul 28 17:18:16.470188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:16.470099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdrjx" event={"ID":"289b8896-51ca-4643-8767-f34ba8c2ab88","Type":"ContainerStarted","Data":"7727cb2b4f8174c2e0d24c10a2d8a82084dfe2ecb093cdcb6f7b16c06d06e938"}
Jul 28 17:18:16.934850 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:16.934812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:16.937456 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:16.937427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecc30941-eb09-4145-8f16-c72d17284331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rrmhq\" (UID: \"ecc30941-eb09-4145-8f16-c72d17284331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:17.094461 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.094428    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"
Jul 28 17:18:17.240548 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.240457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rrmhq"]
Jul 28 17:18:17.243793 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:17.243745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc30941_eb09_4145_8f16_c72d17284331.slice/crio-57ea2c01dbbd7dcc1e00c4e9c4f4fea755b9b2010792f2671891147196ac97ab WatchSource:0}: Error finding container 57ea2c01dbbd7dcc1e00c4e9c4f4fea755b9b2010792f2671891147196ac97ab: Status 404 returned error can't find the container with id 57ea2c01dbbd7dcc1e00c4e9c4f4fea755b9b2010792f2671891147196ac97ab
Jul 28 17:18:17.475170 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.475127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq" event={"ID":"ecc30941-eb09-4145-8f16-c72d17284331","Type":"ContainerStarted","Data":"57ea2c01dbbd7dcc1e00c4e9c4f4fea755b9b2010792f2671891147196ac97ab"}
Jul 28 17:18:17.944379 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.944341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:18:17.944586 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.944391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:18:17.947358 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.947329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:18:17.948358 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.948337    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:18:17.958231 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.958199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa356ac2-7704-422c-adac-5885e9238409-original-pull-secret\") pod \"global-pull-secret-syncer-rw72f\" (UID: \"fa356ac2-7704-422c-adac-5885e9238409\") " pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:18:17.958231 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:17.958214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a33bf6-6706-40de-bdb3-a635e075db13-metrics-certs\") pod \"network-metrics-daemon-7nphk\" (UID: \"13a33bf6-6706-40de-bdb3-a635e075db13\") " pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:18:18.044905 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.044871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:18.047814 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.047785    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:18.058770 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.058744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:18.069667 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.069633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7kz\" (UniqueName: \"kubernetes.io/projected/8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92-kube-api-access-qn7kz\") pod \"network-check-target-sck52\" (UID: \"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92\") " pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:18.211349 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.211262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmkkk\""
Jul 28 17:18:18.216913 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.216884    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lx2z\""
Jul 28 17:18:18.219036 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.219004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nphk"
Jul 28 17:18:18.219151 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.219017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw72f"
Jul 28 17:18:18.225323 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:18.225288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:20.249822 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.249748    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw72f"]
Jul 28 17:18:20.257389 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:20.257341    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa356ac2_7704_422c_adac_5885e9238409.slice/crio-d8b7ca3cc40896c3631618396af9b6cc397da01a74ebf58348a1666bbf8bb499 WatchSource:0}: Error finding container d8b7ca3cc40896c3631618396af9b6cc397da01a74ebf58348a1666bbf8bb499: Status 404 returned error can't find the container with id d8b7ca3cc40896c3631618396af9b6cc397da01a74ebf58348a1666bbf8bb499
Jul 28 17:18:20.470805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.470767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sck52"]
Jul 28 17:18:20.472236 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.472215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7nphk"]
Jul 28 17:18:20.474703 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:20.474669    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e431ceb_b3e3_4d1a_a1b4_1cb71a4bbf92.slice/crio-6b7861bc0e44665a0de5308e1c291b3ff34bf7393c17b9465c373761f56d209d WatchSource:0}: Error finding container 6b7861bc0e44665a0de5308e1c291b3ff34bf7393c17b9465c373761f56d209d: Status 404 returned error can't find the container with id 6b7861bc0e44665a0de5308e1c291b3ff34bf7393c17b9465c373761f56d209d
Jul 28 17:18:20.475417 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:20.475382    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a33bf6_6706_40de_bdb3_a635e075db13.slice/crio-0cada378d9a0b731c0aa95ba491a488ec94457a2c699763bb1568db752337f04 WatchSource:0}: Error finding container 0cada378d9a0b731c0aa95ba491a488ec94457a2c699763bb1568db752337f04: Status 404 returned error can't find the container with id 0cada378d9a0b731c0aa95ba491a488ec94457a2c699763bb1568db752337f04
Jul 28 17:18:20.489841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.489809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sck52" event={"ID":"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92","Type":"ContainerStarted","Data":"6b7861bc0e44665a0de5308e1c291b3ff34bf7393c17b9465c373761f56d209d"}
Jul 28 17:18:20.491052 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.491026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nphk" event={"ID":"13a33bf6-6706-40de-bdb3-a635e075db13","Type":"ContainerStarted","Data":"0cada378d9a0b731c0aa95ba491a488ec94457a2c699763bb1568db752337f04"}
Jul 28 17:18:20.492681 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.492658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq" event={"ID":"ecc30941-eb09-4145-8f16-c72d17284331","Type":"ContainerStarted","Data":"39a220654ff22896039426e532a266a3821e6cf9f31b3b8841cc842e1e428778"}
Jul 28 17:18:20.492791 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.492699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq" event={"ID":"ecc30941-eb09-4145-8f16-c72d17284331","Type":"ContainerStarted","Data":"33869b7c2629b7ea0ce6f29a355a5a34adaa9f1d6015cafe755a9c8294608fa7"}
Jul 28 17:18:20.493730 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.493711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw72f" event={"ID":"fa356ac2-7704-422c-adac-5885e9238409","Type":"ContainerStarted","Data":"d8b7ca3cc40896c3631618396af9b6cc397da01a74ebf58348a1666bbf8bb499"}
Jul 28 17:18:20.495567 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.495544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gdrjx" event={"ID":"289b8896-51ca-4643-8767-f34ba8c2ab88","Type":"ContainerStarted","Data":"1b4ea0e576088ae91c1787ab58c7faf82e6336bc7e116bdcccc0f8948affae75"}
Jul 28 17:18:20.515416 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.515362    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-rrmhq" podStartSLOduration=2.66633869 podStartE2EDuration="5.515345354s" totalImagesPullingTime="2.849006664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:20.514724786 +0000 UTC m=+68.832163672" watchObservedRunningTime="2026-07-28 17:18:20.515345354 +0000 UTC m=+68.832784240"
Jul 28 17:18:20.545240 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:20.545190    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gdrjx" podStartSLOduration=1.229361412 podStartE2EDuration="17.545172778s" totalImagesPullingTime="16.315811366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:20.544425572 +0000 UTC m=+68.861864458" watchObservedRunningTime="2026-07-28 17:18:20.545172778 +0000 UTC m=+68.862611661"
Jul 28 17:18:22.460105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.460073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mh54r"
Jul 28 17:18:22.512620 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.512573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nphk" event={"ID":"13a33bf6-6706-40de-bdb3-a635e075db13","Type":"ContainerStarted","Data":"886e2e215f0f8603eda95a6adb90eeadf24f7736dd6de8f2c88eda80bb7a9883"}
Jul 28 17:18:22.512788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.512688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nphk" event={"ID":"13a33bf6-6706-40de-bdb3-a635e075db13","Type":"ContainerStarted","Data":"12deac1f5d75b38467f84e17d7b404784cc40c8c6edadcc8f4c8d3f7267a23e2"}
Jul 28 17:18:22.533298 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.532735    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7nphk" podStartSLOduration=69.485704915 podStartE2EDuration="1m10.532717521s" totalImagesPullingTime="1.047012606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:22.531621463 +0000 UTC m=+70.849060350" watchObservedRunningTime="2026-07-28 17:18:22.532717521 +0000 UTC m=+70.850156477"
Jul 28 17:18:22.757801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.757695    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-77b8l"]
Jul 28 17:18:22.763227 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.763192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.767061 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.767028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd78l\""
Jul 28 17:18:22.767796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.767771    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:22.768428 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.768408    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:22.768730 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.768706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:22.884675 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-metrics-client-ca\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.884675 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-accelerators-collector-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.884893 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-tls\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.884893 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-sys\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.884893 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.885018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-textfile\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.885018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-root\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.885018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.884973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-wtmp\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.885018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.885005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brhr\" (UniqueName: \"kubernetes.io/projected/99e87093-78bc-46a5-86af-7b8292f8bf38-kube-api-access-6brhr\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.985876 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.985838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-textfile\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.985906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-root\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.985936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-wtmp\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.985963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brhr\" (UniqueName: \"kubernetes.io/projected/99e87093-78bc-46a5-86af-7b8292f8bf38-kube-api-access-6brhr\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-metrics-client-ca\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-root\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-accelerators-collector-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-wtmp\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-tls\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-sys\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-textfile\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e87093-78bc-46a5-86af-7b8292f8bf38-sys\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986718 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-accelerators-collector-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.986908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.986887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e87093-78bc-46a5-86af-7b8292f8bf38-metrics-client-ca\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.988836 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.988816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-tls\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.988937 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.988840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e87093-78bc-46a5-86af-7b8292f8bf38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:22.996445 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:22.996371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brhr\" (UniqueName: \"kubernetes.io/projected/99e87093-78bc-46a5-86af-7b8292f8bf38-kube-api-access-6brhr\") pod \"node-exporter-77b8l\" (UID: \"99e87093-78bc-46a5-86af-7b8292f8bf38\") " pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:23.075932 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:23.075848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77b8l"
Jul 28 17:18:23.501512 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:23.501485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:23.506059 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:23.506032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:24.268577 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.268549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:24.273434 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.273390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:18:24.339344 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.339294    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:24.741921 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.741880    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"]
Jul 28 17:18:24.745639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.745610    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.748132 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.748109    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 17:18:24.748461 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.748440    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 17:18:24.748670 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.748652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wgk9v\""
Jul 28 17:18:24.748790 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.748757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 17:18:24.748864 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.748654    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 17:18:24.749071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.749051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9dpciogu45801\""
Jul 28 17:18:24.749183 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.749170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 17:18:24.757993 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.757958    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"]
Jul 28 17:18:24.802263 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c67b470-8025-42d7-8c47-801a91bc15d9-metrics-client-ca\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-grpc-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngsq\" (UniqueName: \"kubernetes.io/projected/7c67b470-8025-42d7-8c47-801a91bc15d9-kube-api-access-jngsq\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.802708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.802630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-grpc-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngsq\" (UniqueName: \"kubernetes.io/projected/7c67b470-8025-42d7-8c47-801a91bc15d9-kube-api-access-jngsq\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.903457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.903384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c67b470-8025-42d7-8c47-801a91bc15d9-metrics-client-ca\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.904166 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.904114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c67b470-8025-42d7-8c47-801a91bc15d9-metrics-client-ca\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.906546 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.906517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.906674 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.906546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.906674 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.906567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-grpc-tls\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.906996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.906969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.907051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.907039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.907147 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.907127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c67b470-8025-42d7-8c47-801a91bc15d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.912050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:24.912024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngsq\" (UniqueName: \"kubernetes.io/projected/7c67b470-8025-42d7-8c47-801a91bc15d9-kube-api-access-jngsq\") pod \"thanos-querier-5475cdfdb9-8bt6b\" (UID: \"7c67b470-8025-42d7-8c47-801a91bc15d9\") " pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:24.976171 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:24.976141    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e87093_78bc_46a5_86af_7b8292f8bf38.slice/crio-9702e20eaa309481a84ac48f0c29ebd57d8801e642bb7a65b244c56daf351dc5 WatchSource:0}: Error finding container 9702e20eaa309481a84ac48f0c29ebd57d8801e642bb7a65b244c56daf351dc5: Status 404 returned error can't find the container with id 9702e20eaa309481a84ac48f0c29ebd57d8801e642bb7a65b244c56daf351dc5
Jul 28 17:18:25.057290 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.057261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:25.203042 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.203001    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"]
Jul 28 17:18:25.206621 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:25.206592    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c67b470_8025_42d7_8c47_801a91bc15d9.slice/crio-3c848e81a118f917000c7021e9f59f5fa3599fab34b78557441abd669c6fd289 WatchSource:0}: Error finding container 3c848e81a118f917000c7021e9f59f5fa3599fab34b78557441abd669c6fd289: Status 404 returned error can't find the container with id 3c848e81a118f917000c7021e9f59f5fa3599fab34b78557441abd669c6fd289
Jul 28 17:18:25.525938 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.525873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"3c848e81a118f917000c7021e9f59f5fa3599fab34b78557441abd669c6fd289"}
Jul 28 17:18:25.527576 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.527542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77b8l" event={"ID":"99e87093-78bc-46a5-86af-7b8292f8bf38","Type":"ContainerStarted","Data":"9702e20eaa309481a84ac48f0c29ebd57d8801e642bb7a65b244c56daf351dc5"}
Jul 28 17:18:25.529003 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.528974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sck52" event={"ID":"8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92","Type":"ContainerStarted","Data":"447a5324be76d8eec2a3fdb049069efebd5f79c4bba2754ac0e524a89b22c1f7"}
Jul 28 17:18:25.529133 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.529074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:25.530298 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.530280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw72f" event={"ID":"fa356ac2-7704-422c-adac-5885e9238409","Type":"ContainerStarted","Data":"9a23b64d6b35727953a52153114803999b412c1d7171f7ffc0dfed2236a67e51"}
Jul 28 17:18:25.551055 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.551001    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sck52" podStartSLOduration=69.050737835 podStartE2EDuration="1m13.550986341s" totalImagesPullingTime="4.500248506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.550213285 +0000 UTC m=+73.867652173" watchObservedRunningTime="2026-07-28 17:18:25.550986341 +0000 UTC m=+73.868425227"
Jul 28 17:18:25.568659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:25.568603    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rw72f" podStartSLOduration=68.842095669 podStartE2EDuration="1m13.568588183s" totalImagesPullingTime="4.726492514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.568034326 +0000 UTC m=+73.885473215" watchObservedRunningTime="2026-07-28 17:18:25.568588183 +0000 UTC m=+73.886027069"
Jul 28 17:18:26.535753 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:26.535719    2561 generic.go:356] "Generic (PLEG): container finished" podID="99e87093-78bc-46a5-86af-7b8292f8bf38" containerID="2673d2b8e2a19fcd794a34c9a0ba7baef718d69779880ecbe38ab5678c8ae1e0" exitCode=0
Jul 28 17:18:26.536228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:26.535812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77b8l" event={"ID":"99e87093-78bc-46a5-86af-7b8292f8bf38","Type":"ContainerDied","Data":"2673d2b8e2a19fcd794a34c9a0ba7baef718d69779880ecbe38ab5678c8ae1e0"}
Jul 28 17:18:27.157972 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.157888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-848dc66969-kr6xw"]
Jul 28 17:18:27.161015 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.160996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.163589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.163568    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:18:27.163694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.163639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:18:27.163754 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.163727    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6tbdelnvtno6f\""
Jul 28 17:18:27.164662 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.164645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hhf47\""
Jul 28 17:18:27.164662 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.164654    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:18:27.164799 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.164752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:18:27.170720 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.170701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848dc66969-kr6xw"]
Jul 28 17:18:27.224629 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-tls\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-client-certs\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-metrics-server-audit-profiles\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9wp\" (UniqueName: \"kubernetes.io/projected/cdb44e09-af52-4f56-8b19-c663120358b3-kube-api-access-bm9wp\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdb44e09-af52-4f56-8b19-c663120358b3-audit-log\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.224927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.224887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-client-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9wp\" (UniqueName: \"kubernetes.io/projected/cdb44e09-af52-4f56-8b19-c663120358b3-kube-api-access-bm9wp\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdb44e09-af52-4f56-8b19-c663120358b3-audit-log\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-client-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-tls\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-client-certs\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-metrics-server-audit-profiles\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.325979 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.325956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdb44e09-af52-4f56-8b19-c663120358b3-audit-log\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.326339 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.326295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.326824 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.326801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdb44e09-af52-4f56-8b19-c663120358b3-metrics-server-audit-profiles\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.328172 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.328140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-client-ca-bundle\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.328172 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.328164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-tls\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.328289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.328251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdb44e09-af52-4f56-8b19-c663120358b3-secret-metrics-server-client-certs\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.334152 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.334129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9wp\" (UniqueName: \"kubernetes.io/projected/cdb44e09-af52-4f56-8b19-c663120358b3-kube-api-access-bm9wp\") pod \"metrics-server-848dc66969-kr6xw\" (UID: \"cdb44e09-af52-4f56-8b19-c663120358b3\") " pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.431530 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.431490    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"]
Jul 28 17:18:27.436021 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.436000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.438391 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.438369    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:18:27.438673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.438658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqt6d\""
Jul 28 17:18:27.445672 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.445645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"]
Jul 28 17:18:27.470027 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.470002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:27.527071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.527037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgcvs\" (UniqueName: \"kubernetes.io/projected/99794383-ea28-46e3-8b5e-f6e31f485c08-kube-api-access-lgcvs\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.527175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.527131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99794383-ea28-46e3-8b5e-f6e31f485c08-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.542480 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.542453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"f43cd3fc9dedaf0ba565b0b721de3336285170347728b28d50ba2f4ed3cc1ca8"}
Jul 28 17:18:27.542835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.542521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"b51c0771f425432a1bd613ba470df04e03355c8029780564992af0cadf8de013"}
Jul 28 17:18:27.542835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.542559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"59f5c59239b6e6dab6999572b7d906c7b1f1004e96ecee229a542d31944e1f1e"}
Jul 28 17:18:27.544594 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.544571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77b8l" event={"ID":"99e87093-78bc-46a5-86af-7b8292f8bf38","Type":"ContainerStarted","Data":"430ec1fb318feb5812ba92a6b3586bc6a326634fd71561333bea7eec4c3b9651"}
Jul 28 17:18:27.544689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.544613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77b8l" event={"ID":"99e87093-78bc-46a5-86af-7b8292f8bf38","Type":"ContainerStarted","Data":"420486fa95f9c92c578d6832082c47c788e8e8f4fe889e8935fed686f7c7bad8"}
Jul 28 17:18:27.574715 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.574661    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-77b8l" podStartSLOduration=4.675977581 podStartE2EDuration="5.57464368s" totalImagesPullingTime="826.403869ms" totalInitContainerRuntime="72.26223ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:27.574454557 +0000 UTC m=+75.891893499" watchObservedRunningTime="2026-07-28 17:18:27.57464368 +0000 UTC m=+75.892082567"
Jul 28 17:18:27.603714 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.603681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848dc66969-kr6xw"]
Jul 28 17:18:27.606844 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:27.606812    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb44e09_af52_4f56_8b19_c663120358b3.slice/crio-d15dbc8d2ced38875d1b961e4ce2b9b4dffadb44d58d1f93790017311ff98e36 WatchSource:0}: Error finding container d15dbc8d2ced38875d1b961e4ce2b9b4dffadb44d58d1f93790017311ff98e36: Status 404 returned error can't find the container with id d15dbc8d2ced38875d1b961e4ce2b9b4dffadb44d58d1f93790017311ff98e36
Jul 28 17:18:27.628451 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.628422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99794383-ea28-46e3-8b5e-f6e31f485c08-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.628575 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.628505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcvs\" (UniqueName: \"kubernetes.io/projected/99794383-ea28-46e3-8b5e-f6e31f485c08-kube-api-access-lgcvs\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.630815 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.630796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/99794383-ea28-46e3-8b5e-f6e31f485c08-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.644662 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.644636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcvs\" (UniqueName: \"kubernetes.io/projected/99794383-ea28-46e3-8b5e-f6e31f485c08-kube-api-access-lgcvs\") pod \"monitoring-plugin-68b7fc9574-ggjbt\" (UID: \"99794383-ea28-46e3-8b5e-f6e31f485c08\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.745612 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.745531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:27.895096 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:27.894944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"]
Jul 28 17:18:28.013124 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:28.013041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99794383_ea28_46e3_8b5e_f6e31f485c08.slice/crio-dd1e489cc2f67efe3c804c3614d295771b637f22a58328d44f6db69cdeb2cddb WatchSource:0}: Error finding container dd1e489cc2f67efe3c804c3614d295771b637f22a58328d44f6db69cdeb2cddb: Status 404 returned error can't find the container with id dd1e489cc2f67efe3c804c3614d295771b637f22a58328d44f6db69cdeb2cddb
Jul 28 17:18:28.243836 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.243807    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:18:28.247105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.247082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.257977 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.257926    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:18:28.333985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.333901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwsr\" (UniqueName: \"kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.333985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.333957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.334221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.334010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.334221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.334035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.334221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.334094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.334221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.334109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.334221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.334130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.434776 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.434741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.434776 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.434780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.434998 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.434962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435058 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435058 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435166 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435166 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwsr\" (UniqueName: \"kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.435780 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.435758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.436025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.436002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.436139 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.436120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.438065 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.438036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.438196 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.438179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.443532 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.443514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwsr\" (UniqueName: \"kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr\") pod \"console-69f576c94d-ltc49\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") " pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.557498 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.557459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"89503444e72d38dd02478030334946a600fc02e2ab205b58a4eb483a97f8ff19"}
Jul 28 17:18:28.557929 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.557545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"b55eeed05b3a14f8638bbaa165bd1ec0e175c207ce7c78bf29ab0be01140992e"}
Jul 28 17:18:28.557929 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.557587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" event={"ID":"7c67b470-8025-42d7-8c47-801a91bc15d9","Type":"ContainerStarted","Data":"66f8d851f657145d69955abc187c4b32c1854fdb1cb02d2e72c28c48ebeccbef"}
Jul 28 17:18:28.557929 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.557763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:28.558054 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.558019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:28.562650 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.562571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt" event={"ID":"99794383-ea28-46e3-8b5e-f6e31f485c08","Type":"ContainerStarted","Data":"dd1e489cc2f67efe3c804c3614d295771b637f22a58328d44f6db69cdeb2cddb"}
Jul 28 17:18:28.570521 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.570491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw" event={"ID":"cdb44e09-af52-4f56-8b19-c663120358b3","Type":"ContainerStarted","Data":"d15dbc8d2ced38875d1b961e4ce2b9b4dffadb44d58d1f93790017311ff98e36"}
Jul 28 17:18:28.584747 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.584610    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b" podStartSLOduration=1.9259319449999999 podStartE2EDuration="4.584588032s" totalImagesPullingTime="2.658656087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:28.583716304 +0000 UTC m=+76.901155228" watchObservedRunningTime="2026-07-28 17:18:28.584588032 +0000 UTC m=+76.902026919"
Jul 28 17:18:28.750567 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:28.750509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:18:29.025123 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.025077    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:29.031947 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.031884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.034722 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.034694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:18:29.038245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.037924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:18:29.038245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.037970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:18:29.038245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.037924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:18:29.038245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.037929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:18:29.038245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.038089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:18:29.039460 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.039089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:18:29.039595 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.039544    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k8gmf\""
Jul 28 17:18:29.039879 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.039655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:18:29.039879 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.039792    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:18:29.040286 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.040089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:18:29.040286 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.040179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:18:29.040524 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.040506    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:18:29.040742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.040721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fbaui4ks4aajn\""
Jul 28 17:18:29.044076 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.043232    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:18:29.046472 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.046452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:18:29.058278 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.058253    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:29.145362 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.145848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jt28\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-kube-api-access-2jt28\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.145959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.146008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.146061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.146351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.146101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246791 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246791 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246791 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246791 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jt28\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-kube-api-access-2jt28\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.246914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.246994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247840 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.247950 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.247841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.248458 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.248326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.249560 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.249538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.250019 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.249999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.250838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.250700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.250838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.250729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.250838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.250810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.251686 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.251660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.252399 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.252372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.253659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.253613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.253743 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.253722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c0df93a-7f35-4538-bbc2-19323866a040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.253808 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.253750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.254266 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.254241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.255199 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.255156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.256356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.255872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.256356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.256006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-config\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.256356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.256294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c0df93a-7f35-4538-bbc2-19323866a040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.258097 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.258054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jt28\" (UniqueName: \"kubernetes.io/projected/6c0df93a-7f35-4538-bbc2-19323866a040-kube-api-access-2jt28\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.258382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.258365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c0df93a-7f35-4538-bbc2-19323866a040-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c0df93a-7f35-4538-bbc2-19323866a040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.348992 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.348911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:29.576289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.576257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f576c94d-ltc49" event={"ID":"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b","Type":"ContainerStarted","Data":"829757eae5646076f2122c3d796c151bf7220a2ad822a68f4a877b72dd084694"}
Jul 28 17:18:29.852224 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:29.852192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:29.855566 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:29.855536    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0df93a_7f35_4538_bbc2_19323866a040.slice/crio-657bdcd7bc0e2fd582d82223c4d61dd2a1ba573f242de7ae3145cae4aef01a01 WatchSource:0}: Error finding container 657bdcd7bc0e2fd582d82223c4d61dd2a1ba573f242de7ae3145cae4aef01a01: Status 404 returned error can't find the container with id 657bdcd7bc0e2fd582d82223c4d61dd2a1ba573f242de7ae3145cae4aef01a01
Jul 28 17:18:30.584150 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.584102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt" event={"ID":"99794383-ea28-46e3-8b5e-f6e31f485c08","Type":"ContainerStarted","Data":"721e9713f0b8a45e26b6c92608df395c839c184a0f258b4d122b1f45ce6991d0"}
Jul 28 17:18:30.584616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.584577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:30.585969 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.585940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw" event={"ID":"cdb44e09-af52-4f56-8b19-c663120358b3","Type":"ContainerStarted","Data":"d21c4e2deb3aeb7d4f89fdc6d3569628edc5a184c8c994359e4930f76482d0f4"}
Jul 28 17:18:30.587834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.587810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f576c94d-ltc49" event={"ID":"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b","Type":"ContainerStarted","Data":"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"}
Jul 28 17:18:30.589291 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.589255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"657bdcd7bc0e2fd582d82223c4d61dd2a1ba573f242de7ae3145cae4aef01a01"}
Jul 28 17:18:30.590348 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.590327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt"
Jul 28 17:18:30.605861 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.605804    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ggjbt" podStartSLOduration=1.923893318 podStartE2EDuration="3.605788099s" totalImagesPullingTime="1.681894781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:30.604387272 +0000 UTC m=+78.921826154" watchObservedRunningTime="2026-07-28 17:18:30.605788099 +0000 UTC m=+78.923226986"
Jul 28 17:18:30.627942 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.627892    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw" podStartSLOduration=1.543945028 podStartE2EDuration="3.627870631s" totalImagesPullingTime="2.083925603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:30.627749242 +0000 UTC m=+78.945188128" watchObservedRunningTime="2026-07-28 17:18:30.627870631 +0000 UTC m=+78.945309524"
Jul 28 17:18:30.670837 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:30.670771    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69f576c94d-ltc49" podStartSLOduration=2.670751759 podStartE2EDuration="2.670751759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:30.67023053 +0000 UTC m=+78.987669417" watchObservedRunningTime="2026-07-28 17:18:30.670751759 +0000 UTC m=+78.988190647"
Jul 28 17:18:31.594200 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.594158    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c0df93a-7f35-4538-bbc2-19323866a040" containerID="a189900900d18b37aa87d195b19bdb18840e0bffc5a599691e32334969265a84" exitCode=0
Jul 28 17:18:31.594625 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.594281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerDied","Data":"a189900900d18b37aa87d195b19bdb18840e0bffc5a599691e32334969265a84"}
Jul 28 17:18:31.858002 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.857916    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:18:31.908267 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.908232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:18:31.910968 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.910949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:31.927060 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:31.927033    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:18:32.073645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.073852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.073852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.073852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfndq\" (UniqueName: \"kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.073852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.074025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.074025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.073912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175260 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175260 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfndq\" (UniqueName: \"kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.175506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.175472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.176265 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.176223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.176265 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.176251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.176421 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.176328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.176421 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.176364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.177807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.177789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.178230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.178213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.186114 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.186092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfndq\" (UniqueName: \"kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq\") pod \"console-5bdbc444b8-9s7r7\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") " pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.220508 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.220472    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:32.386408 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.386327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:18:32.390240 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:18:32.390202    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3190874c_7965_4970_af5b_ca17ab080d8c.slice/crio-95f0da6717072b2694eb4aa50b361512f75efcee9e7aaad6642c448ad5ee01a0 WatchSource:0}: Error finding container 95f0da6717072b2694eb4aa50b361512f75efcee9e7aaad6642c448ad5ee01a0: Status 404 returned error can't find the container with id 95f0da6717072b2694eb4aa50b361512f75efcee9e7aaad6642c448ad5ee01a0
Jul 28 17:18:32.600746 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.600657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbc444b8-9s7r7" event={"ID":"3190874c-7965-4970-af5b-ca17ab080d8c","Type":"ContainerStarted","Data":"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"}
Jul 28 17:18:32.600746 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.600721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbc444b8-9s7r7" event={"ID":"3190874c-7965-4970-af5b-ca17ab080d8c","Type":"ContainerStarted","Data":"95f0da6717072b2694eb4aa50b361512f75efcee9e7aaad6642c448ad5ee01a0"}
Jul 28 17:18:32.628732 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:32.628675    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bdbc444b8-9s7r7" podStartSLOduration=1.628659545 podStartE2EDuration="1.628659545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:32.627098404 +0000 UTC m=+80.944537290" watchObservedRunningTime="2026-07-28 17:18:32.628659545 +0000 UTC m=+80.946098431"
Jul 28 17:18:35.595516 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.595489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5475cdfdb9-8bt6b"
Jul 28 17:18:35.617195 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"a90986e32633be864925bceb6918ed9aee381017969c9ba2ef53e0174a1071dc"}
Jul 28 17:18:35.617376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"01a76bacea60d81b73c722af19340dcaee4835bd573cb33f2f07360580b89139"}
Jul 28 17:18:35.617376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"60a927f45ef2ad3fdd1e7f5287bc2f0b6526ef2cf690fd7f823bc78658f43119"}
Jul 28 17:18:35.617376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"f64dfa034967e510f5dd1f2f44a3e2d8819be4ed6ce5281f1918f8433e99f38b"}
Jul 28 17:18:35.617376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"0fd61093d5fba9802d6928b2a148e2da3f217887af32ca279f0e6970dbc6b948"}
Jul 28 17:18:35.617376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.617367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c0df93a-7f35-4538-bbc2-19323866a040","Type":"ContainerStarted","Data":"7ce87d0c9664dfc1a9802bc7d11c750cbf14918fbc078774fc12d3aaed70da7a"}
Jul 28 17:18:35.659384 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:35.659301    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.4905704699999998 podStartE2EDuration="7.659278007s" totalImagesPullingTime="4.150055841s" totalInitContainerRuntime="18.651696ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:35.656832776 +0000 UTC m=+83.974271664" watchObservedRunningTime="2026-07-28 17:18:35.659278007 +0000 UTC m=+83.976716895"
Jul 28 17:18:38.558354 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:38.558284    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:39.349126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:39.349094    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:42.220717 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:42.220685    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:42.221205 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:42.220738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:42.227789 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:42.227765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:42.650574 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:42.650501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:18:42.708750 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:42.708711    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:18:47.470376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:47.470329    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:47.470855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:47.470402    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:18:49.542058 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:49.541993    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-584c5d6559-8w77b" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerName="console" containerID="cri-o://e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54" gracePeriod=15
Jul 28 17:18:50.358078 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.358054    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:50.442078 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.441991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442078 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442078 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442096    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4xsz\" (UniqueName: \"kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca\") pod \"5e73fec9-2e94-44ee-82f0-22ebc8488634\" (UID: \"5e73fec9-2e94-44ee-82f0-22ebc8488634\") "
Jul 28 17:18:50.442629 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config" (OuterVolumeSpecName: "console-config") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:50.442720 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:50.442720 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.442686    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca" (OuterVolumeSpecName: "service-ca") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:50.444505 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.444482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:50.444586 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.444542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:50.444586 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.444540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz" (OuterVolumeSpecName: "kube-api-access-j4xsz") pod "5e73fec9-2e94-44ee-82f0-22ebc8488634" (UID: "5e73fec9-2e94-44ee-82f0-22ebc8488634"). InnerVolumeSpecName "kube-api-access-j4xsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:50.543228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543196    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.543228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543222    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-oauth-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.543228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543233    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xsz\" (UniqueName: \"kubernetes.io/projected/5e73fec9-2e94-44ee-82f0-22ebc8488634-kube-api-access-j4xsz\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.543665 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543243    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.543665 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543251    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e73fec9-2e94-44ee-82f0-22ebc8488634-service-ca\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.543665 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.543259    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e73fec9-2e94-44ee-82f0-22ebc8488634-console-oauth-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:50.676965 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.676929    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerID="e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54" exitCode=0
Jul 28 17:18:50.677150 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.676971    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584c5d6559-8w77b"
Jul 28 17:18:50.677150 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.676989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584c5d6559-8w77b" event={"ID":"5e73fec9-2e94-44ee-82f0-22ebc8488634","Type":"ContainerDied","Data":"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"}
Jul 28 17:18:50.677150 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.677034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584c5d6559-8w77b" event={"ID":"5e73fec9-2e94-44ee-82f0-22ebc8488634","Type":"ContainerDied","Data":"25d6c43363ca6bc918741c5576a2c4f6e48b4cb7241544c61c23cf2aa965cfd3"}
Jul 28 17:18:50.677150 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.677056    2561 scope.go:117] "RemoveContainer" containerID="e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"
Jul 28 17:18:50.685894 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.685874    2561 scope.go:117] "RemoveContainer" containerID="e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"
Jul 28 17:18:50.686125 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:50.686105    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54\": container with ID starting with e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54 not found: ID does not exist" containerID="e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"
Jul 28 17:18:50.686177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.686134    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54"} err="failed to get container status \"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54\": rpc error: code = NotFound desc = could not find container \"e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54\": container with ID starting with e6c8baf5d4c42b2380b5ff3f75fa6990d0c402338225cc7b2dc994e868652b54 not found: ID does not exist"
Jul 28 17:18:50.699834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.699782    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:50.706315 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:50.706280    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-584c5d6559-8w77b"]
Jul 28 17:18:52.202717 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:52.202667    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" path="/var/lib/kubelet/pods/5e73fec9-2e94-44ee-82f0-22ebc8488634/volumes"
Jul 28 17:18:56.538380 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:56.538348    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sck52"
Jul 28 17:18:57.624397 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:57.624340    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69f576c94d-ltc49" podUID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" containerName="console" containerID="cri-o://46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db" gracePeriod=15
Jul 28 17:18:57.899194 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:57.899132    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:58.003833 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003883    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.003997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqwsr\" (UniqueName: \"kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004276 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.004018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert\") pod \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\" (UID: \"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b\") "
Jul 28 17:18:58.004364 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.004280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config" (OuterVolumeSpecName: "console-config") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:58.004443 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.004417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:58.004495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.004445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca" (OuterVolumeSpecName: "service-ca") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:58.004529 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.004507    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:58.005992 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.005957    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:58.006115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.006097    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr" (OuterVolumeSpecName: "kube-api-access-hqwsr") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "kube-api-access-hqwsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:58.006151 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.006106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" (UID: "db9b9bbe-0a14-4a1e-968a-8b07f2a5902b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:58.105234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105204    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-oauth-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105228    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-trusted-ca-bundle\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105237    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105246    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-service-ca\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105254    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-oauth-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105261    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwsr\" (UniqueName: \"kubernetes.io/projected/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-kube-api-access-hqwsr\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.105471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.105270    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b-console-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:18:58.710363 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.710327    2561 generic.go:356] "Generic (PLEG): container finished" podID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" containerID="46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db" exitCode=0
Jul 28 17:18:58.710363 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.710366    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f576c94d-ltc49"
Jul 28 17:18:58.710777 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.710403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f576c94d-ltc49" event={"ID":"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b","Type":"ContainerDied","Data":"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"}
Jul 28 17:18:58.710777 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.710456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f576c94d-ltc49" event={"ID":"db9b9bbe-0a14-4a1e-968a-8b07f2a5902b","Type":"ContainerDied","Data":"829757eae5646076f2122c3d796c151bf7220a2ad822a68f4a877b72dd084694"}
Jul 28 17:18:58.710777 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.710481    2561 scope.go:117] "RemoveContainer" containerID="46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"
Jul 28 17:18:58.718323 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.718291    2561 scope.go:117] "RemoveContainer" containerID="46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"
Jul 28 17:18:58.718578 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:18:58.718560    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db\": container with ID starting with 46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db not found: ID does not exist" containerID="46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"
Jul 28 17:18:58.718636 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.718586    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db"} err="failed to get container status \"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db\": rpc error: code = NotFound desc = could not find container \"46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db\": container with ID starting with 46cd576977be9ac5e87b6f70e9eeda4a4ca9939467b36fa469b28a45dc0556db not found: ID does not exist"
Jul 28 17:18:58.727382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.727360    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:18:58.731151 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:18:58.731130    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69f576c94d-ltc49"]
Jul 28 17:19:00.203592 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:00.203559    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" path="/var/lib/kubelet/pods/db9b9bbe-0a14-4a1e-968a-8b07f2a5902b/volumes"
Jul 28 17:19:07.475928 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:07.475902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:19:07.479834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:07.479811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-848dc66969-kr6xw"
Jul 28 17:19:07.733974 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:07.733867    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67564fb4b-2s27v" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" containerName="console" containerID="cri-o://08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995" gracePeriod=15
Jul 28 17:19:07.975663 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:07.975640    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:19:08.087482 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087482 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087482 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087480    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqfrp\" (UniqueName: \"kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config\") pod \"e0832de7-95d2-409e-8644-f86e0d49a297\" (UID: \"e0832de7-95d2-409e-8644-f86e0d49a297\") "
Jul 28 17:19:08.087931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:08.087931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087900    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-trusted-ca-bundle\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.087931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:08.087931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.087920    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca" (OuterVolumeSpecName: "service-ca") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:08.088131 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.088099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config" (OuterVolumeSpecName: "console-config") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:08.089796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.089766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp" (OuterVolumeSpecName: "kube-api-access-fqfrp") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "kube-api-access-fqfrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:08.089796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.089764    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:08.089944 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.089800    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e0832de7-95d2-409e-8644-f86e0d49a297" (UID: "e0832de7-95d2-409e-8644-f86e0d49a297"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:08.188418 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188386    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-oauth-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.188418 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188415    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0832de7-95d2-409e-8644-f86e0d49a297-console-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.188418 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188424    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-console-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.188643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188435    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-oauth-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.188643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188443    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0832de7-95d2-409e-8644-f86e0d49a297-service-ca\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.188643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.188451    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfrp\" (UniqueName: \"kubernetes.io/projected/e0832de7-95d2-409e-8644-f86e0d49a297-kube-api-access-fqfrp\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:19:08.748036 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.747999    2561 generic.go:356] "Generic (PLEG): container finished" podID="e0832de7-95d2-409e-8644-f86e0d49a297" containerID="08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995" exitCode=0
Jul 28 17:19:08.748036 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.748039    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67564fb4b-2s27v"
Jul 28 17:19:08.748537 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.748066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67564fb4b-2s27v" event={"ID":"e0832de7-95d2-409e-8644-f86e0d49a297","Type":"ContainerDied","Data":"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"}
Jul 28 17:19:08.748537 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.748104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67564fb4b-2s27v" event={"ID":"e0832de7-95d2-409e-8644-f86e0d49a297","Type":"ContainerDied","Data":"2f7221d0718edb4f6158cbb69ed3dc64e521a70ac9a50862c8d7d48f8f694606"}
Jul 28 17:19:08.748537 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.748123    2561 scope.go:117] "RemoveContainer" containerID="08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"
Jul 28 17:19:08.756130 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.756115    2561 scope.go:117] "RemoveContainer" containerID="08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"
Jul 28 17:19:08.756400 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:19:08.756383    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995\": container with ID starting with 08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995 not found: ID does not exist" containerID="08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"
Jul 28 17:19:08.756465 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.756408    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995"} err="failed to get container status \"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995\": rpc error: code = NotFound desc = could not find container \"08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995\": container with ID starting with 08c76f1e3c0682dd945baee311551156736c8862c5477cb19a95b854579ac995 not found: ID does not exist"
Jul 28 17:19:08.768025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.764875    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:19:08.769796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:08.769774    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67564fb4b-2s27v"]
Jul 28 17:19:10.202639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:10.202605    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" path="/var/lib/kubelet/pods/e0832de7-95d2-409e-8644-f86e0d49a297/volumes"
Jul 28 17:19:29.349732 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:29.349702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:29.367254 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:29.367226    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:29.839216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:29.839188    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:45.402604 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402854    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402867    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402876    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402881    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402893    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402898    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402938    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e73fec9-2e94-44ee-82f0-22ebc8488634" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402968    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0832de7-95d2-409e-8644-f86e0d49a297" containerName="console"
Jul 28 17:19:45.403107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.402977    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9b9bbe-0a14-4a1e-968a-8b07f2a5902b" containerName="console"
Jul 28 17:19:45.405184 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.405168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.415480 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.415454    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:19:45.488344 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488344 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26jj\" (UniqueName: \"kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.488640 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.488533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589774 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26jj\" (UniqueName: \"kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589774 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589774 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589774 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.589774 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.589733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.590436 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.590409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.590436 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.590419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.590635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.590515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.590635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.590515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.592701 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.592675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.592701 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.592683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.599227 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.599207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26jj\" (UniqueName: \"kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj\") pod \"console-679b96ff76-djzpb\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") " pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.714498 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.714457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:45.835293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.835264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:19:45.838165 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:19:45.838142    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364c70c_526b_4734_9671_18f59649456a.slice/crio-9f59a994abfcd7c71ad976efa2db4b4e598b4f1fd069db00f4e7875e0d618d8b WatchSource:0}: Error finding container 9f59a994abfcd7c71ad976efa2db4b4e598b4f1fd069db00f4e7875e0d618d8b: Status 404 returned error can't find the container with id 9f59a994abfcd7c71ad976efa2db4b4e598b4f1fd069db00f4e7875e0d618d8b
Jul 28 17:19:45.883447 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:45.883419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b96ff76-djzpb" event={"ID":"b364c70c-526b-4734-9671-18f59649456a","Type":"ContainerStarted","Data":"9f59a994abfcd7c71ad976efa2db4b4e598b4f1fd069db00f4e7875e0d618d8b"}
Jul 28 17:19:46.888938 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:46.888907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b96ff76-djzpb" event={"ID":"b364c70c-526b-4734-9671-18f59649456a","Type":"ContainerStarted","Data":"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"}
Jul 28 17:19:46.909524 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:46.909471    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679b96ff76-djzpb" podStartSLOduration=1.909456171 podStartE2EDuration="1.909456171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:46.908299437 +0000 UTC m=+155.225738325" watchObservedRunningTime="2026-07-28 17:19:46.909456171 +0000 UTC m=+155.226895057"
Jul 28 17:19:55.715339 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:55.715282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:55.715813 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:55.715363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:55.720105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:55.720083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:55.925458 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:55.925431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:19:55.994321 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:19:55.994216    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:20:21.013070 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.013015    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bdbc444b8-9s7r7" podUID="3190874c-7965-4970-af5b-ca17ab080d8c" containerName="console" containerID="cri-o://640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d" gracePeriod=15
Jul 28 17:20:21.241659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.241638    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:20:21.406180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfndq\" (UniqueName: \"kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406322    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config\") pod \"3190874c-7965-4970-af5b-ca17ab080d8c\" (UID: \"3190874c-7965-4970-af5b-ca17ab080d8c\") "
Jul 28 17:20:21.406683 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406573    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:21.406683 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:21.406902 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.406846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config" (OuterVolumeSpecName: "console-config") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:21.407061 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.407036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca" (OuterVolumeSpecName: "service-ca") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:21.408419 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.408398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:21.408509 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.408487    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq" (OuterVolumeSpecName: "kube-api-access-lfndq") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "kube-api-access-lfndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:21.408886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.408860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3190874c-7965-4970-af5b-ca17ab080d8c" (UID: "3190874c-7965-4970-af5b-ca17ab080d8c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:21.507268 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507238    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfndq\" (UniqueName: \"kubernetes.io/projected/3190874c-7965-4970-af5b-ca17ab080d8c-kube-api-access-lfndq\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507268 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507264    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-console-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507268 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507272    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-oauth-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507280    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-oauth-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507289    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-trusted-ca-bundle\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507297    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3190874c-7965-4970-af5b-ca17ab080d8c-console-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:21.507466 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:21.507320    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3190874c-7965-4970-af5b-ca17ab080d8c-service-ca\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:20:22.011136 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.011102    2561 generic.go:356] "Generic (PLEG): container finished" podID="3190874c-7965-4970-af5b-ca17ab080d8c" containerID="640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d" exitCode=0
Jul 28 17:20:22.011301 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.011145    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbc444b8-9s7r7"
Jul 28 17:20:22.011301 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.011173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbc444b8-9s7r7" event={"ID":"3190874c-7965-4970-af5b-ca17ab080d8c","Type":"ContainerDied","Data":"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"}
Jul 28 17:20:22.011301 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.011214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbc444b8-9s7r7" event={"ID":"3190874c-7965-4970-af5b-ca17ab080d8c","Type":"ContainerDied","Data":"95f0da6717072b2694eb4aa50b361512f75efcee9e7aaad6642c448ad5ee01a0"}
Jul 28 17:20:22.011301 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.011239    2561 scope.go:117] "RemoveContainer" containerID="640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"
Jul 28 17:20:22.019862 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.019736    2561 scope.go:117] "RemoveContainer" containerID="640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"
Jul 28 17:20:22.020051 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:20:22.019991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d\": container with ID starting with 640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d not found: ID does not exist" containerID="640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"
Jul 28 17:20:22.020051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.020011    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d"} err="failed to get container status \"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d\": rpc error: code = NotFound desc = could not find container \"640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d\": container with ID starting with 640da9a1cd30a02f67c334672a6d9361eb6d535f9135193381afa90a9926450d not found: ID does not exist"
Jul 28 17:20:22.032547 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.032528    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:20:22.036264 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.036245    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdbc444b8-9s7r7"]
Jul 28 17:20:22.206620 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:20:22.206587    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3190874c-7965-4970-af5b-ca17ab080d8c" path="/var/lib/kubelet/pods/3190874c-7965-4970-af5b-ca17ab080d8c/volumes"
Jul 28 17:22:12.032505 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:12.032477    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:22:43.858589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.858559    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-8mhbf"]
Jul 28 17:22:43.861019 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.858869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3190874c-7965-4970-af5b-ca17ab080d8c" containerName="console"
Jul 28 17:22:43.861019 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.858881    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3190874c-7965-4970-af5b-ca17ab080d8c" containerName="console"
Jul 28 17:22:43.861019 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.858938    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3190874c-7965-4970-af5b-ca17ab080d8c" containerName="console"
Jul 28 17:22:43.861895 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.861881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:43.864710 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.864686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:22:43.864710 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.864700    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:22:43.864899 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.864700    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9crzv\""
Jul 28 17:22:43.864899 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.864700    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:22:43.874050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.874031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8mhbf"]
Jul 28 17:22:43.936944 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.936905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/adca4fbe-b95e-4329-8b91-00cec369bf6d-data\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:43.937120 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:43.936964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hn9\" (UniqueName: \"kubernetes.io/projected/adca4fbe-b95e-4329-8b91-00cec369bf6d-kube-api-access-86hn9\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.038240 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.038199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hn9\" (UniqueName: \"kubernetes.io/projected/adca4fbe-b95e-4329-8b91-00cec369bf6d-kube-api-access-86hn9\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.038434 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.038269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/adca4fbe-b95e-4329-8b91-00cec369bf6d-data\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.038602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.038586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/adca4fbe-b95e-4329-8b91-00cec369bf6d-data\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.049985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.049967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hn9\" (UniqueName: \"kubernetes.io/projected/adca4fbe-b95e-4329-8b91-00cec369bf6d-kube-api-access-86hn9\") pod \"seaweedfs-64568bcd49-8mhbf\" (UID: \"adca4fbe-b95e-4329-8b91-00cec369bf6d\") " pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.171000 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.170965    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:44.300157 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.300135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8mhbf"]
Jul 28 17:22:44.302492 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:22:44.302465    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadca4fbe_b95e_4329_8b91_00cec369bf6d.slice/crio-4cdab29498998752fbff68667b9e84b520b8ff47fc8cf200d0fc47a9153ee6f5 WatchSource:0}: Error finding container 4cdab29498998752fbff68667b9e84b520b8ff47fc8cf200d0fc47a9153ee6f5: Status 404 returned error can't find the container with id 4cdab29498998752fbff68667b9e84b520b8ff47fc8cf200d0fc47a9153ee6f5
Jul 28 17:22:44.304034 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.304020    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:22:44.497472 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:44.497392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8mhbf" event={"ID":"adca4fbe-b95e-4329-8b91-00cec369bf6d","Type":"ContainerStarted","Data":"4cdab29498998752fbff68667b9e84b520b8ff47fc8cf200d0fc47a9153ee6f5"}
Jul 28 17:22:47.510190 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:47.510160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8mhbf" event={"ID":"adca4fbe-b95e-4329-8b91-00cec369bf6d","Type":"ContainerStarted","Data":"91352eb7e192a0f7486f7cab0ba285868b22c3dfa45acafbce0a1f33e8cd3a63"}
Jul 28 17:22:47.510590 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:47.510271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:22:47.527153 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:47.527113    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-8mhbf" podStartSLOduration=1.48736025 podStartE2EDuration="4.527096466s" totalImagesPullingTime="3.039736216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:47.526975703 +0000 UTC m=+335.844414590" watchObservedRunningTime="2026-07-28 17:22:47.527096466 +0000 UTC m=+335.844535354"
Jul 28 17:22:53.515365 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:22:53.515333    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-8mhbf"
Jul 28 17:23:41.861044 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:41.861012    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb94f5997-w2qrm"]
Jul 28 17:23:41.863138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:41.863121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:41.899733 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:41.899710    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb94f5997-w2qrm"]
Jul 28 17:23:42.023745 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.023711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.023927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.023752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.023927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.023774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-oauth-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.023927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.023891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-trusted-ca-bundle\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.024074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.023970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-service-ca\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.024074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.024005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjfw\" (UniqueName: \"kubernetes.io/projected/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-kube-api-access-9hjfw\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.024074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.024053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-oauth-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.124957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.124999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-oauth-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125295 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-trusted-ca-bundle\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125295 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-service-ca\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125295 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjfw\" (UniqueName: \"kubernetes.io/projected/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-kube-api-access-9hjfw\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125467 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-oauth-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125867 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.125867 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-service-ca\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.126043 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.125843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-oauth-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.126100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.126080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-trusted-ca-bundle\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.127627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.127609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-oauth-config\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.127719 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.127692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-console-serving-cert\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.134503 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.134479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjfw\" (UniqueName: \"kubernetes.io/projected/b7e0b011-8ae8-4c68-b14d-e2153ff229fa-kube-api-access-9hjfw\") pod \"console-7bb94f5997-w2qrm\" (UID: \"b7e0b011-8ae8-4c68-b14d-e2153ff229fa\") " pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.171136 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.171111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:42.300871 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.300784    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb94f5997-w2qrm"]
Jul 28 17:23:42.303088 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:23:42.303062    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e0b011_8ae8_4c68_b14d_e2153ff229fa.slice/crio-81647703a09fb7a5bee6c5fe8f5ff20734079795888459cc4be79a038c7820c9 WatchSource:0}: Error finding container 81647703a09fb7a5bee6c5fe8f5ff20734079795888459cc4be79a038c7820c9: Status 404 returned error can't find the container with id 81647703a09fb7a5bee6c5fe8f5ff20734079795888459cc4be79a038c7820c9
Jul 28 17:23:42.706027 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.705986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb94f5997-w2qrm" event={"ID":"b7e0b011-8ae8-4c68-b14d-e2153ff229fa","Type":"ContainerStarted","Data":"1242ce86d143c427d496e3861a9491886e3576c0b9fb519ef97402471ed51b79"}
Jul 28 17:23:42.706221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.706038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb94f5997-w2qrm" event={"ID":"b7e0b011-8ae8-4c68-b14d-e2153ff229fa","Type":"ContainerStarted","Data":"81647703a09fb7a5bee6c5fe8f5ff20734079795888459cc4be79a038c7820c9"}
Jul 28 17:23:42.724448 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:42.724392    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb94f5997-w2qrm" podStartSLOduration=1.724372647 podStartE2EDuration="1.724372647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:42.724015348 +0000 UTC m=+391.041454235" watchObservedRunningTime="2026-07-28 17:23:42.724372647 +0000 UTC m=+391.041811535"
Jul 28 17:23:52.171275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:52.171243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:52.171761 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:52.171350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:52.176491 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:52.176469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:52.746581 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:52.746555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb94f5997-w2qrm"
Jul 28 17:23:52.810401 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:23:52.810371    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:24:03.326446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.326412    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-z7wgb"]
Jul 28 17:24:03.328535 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.328521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.331168 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.331145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:24:03.331168 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.331157    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-m488z\""
Jul 28 17:24:03.351756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.351730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z7wgb"]
Jul 28 17:24:03.352837 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.352813    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vbjnd"]
Jul 28 17:24:03.355076 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.355061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.357704 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.357687    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:24:03.357802 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.357752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zkptp\""
Jul 28 17:24:03.368414 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.368391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vbjnd"]
Jul 28 17:24:03.391923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.391898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcdl\" (UniqueName: \"kubernetes.io/projected/8cd4b27a-a718-4089-8a02-162bb1fda220-kube-api-access-wfcdl\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.392064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.392000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b27a-a718-4089-8a02-162bb1fda220-tls-certs\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.392064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.392030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-cert\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.392064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.392051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvh6\" (UniqueName: \"kubernetes.io/projected/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-kube-api-access-prvh6\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.492523 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.492496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcdl\" (UniqueName: \"kubernetes.io/projected/8cd4b27a-a718-4089-8a02-162bb1fda220-kube-api-access-wfcdl\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.492679 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.492563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b27a-a718-4089-8a02-162bb1fda220-tls-certs\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.492679 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.492588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-cert\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.492679 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.492606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvh6\" (UniqueName: \"kubernetes.io/projected/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-kube-api-access-prvh6\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.495005 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.494971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b27a-a718-4089-8a02-162bb1fda220-tls-certs\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.495133 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.495036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-cert\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.501219 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.501198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvh6\" (UniqueName: \"kubernetes.io/projected/cc2143ac-9761-4e73-a1bb-38d7d84ce1b0-kube-api-access-prvh6\") pod \"odh-model-controller-75d9cdbddf-vbjnd\" (UID: \"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0\") " pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.501776 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.501753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcdl\" (UniqueName: \"kubernetes.io/projected/8cd4b27a-a718-4089-8a02-162bb1fda220-kube-api-access-wfcdl\") pod \"model-serving-api-7b954ffc95-z7wgb\" (UID: \"8cd4b27a-a718-4089-8a02-162bb1fda220\") " pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.638921 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.638844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:03.665015 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.664989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:03.784929 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.784900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z7wgb"]
Jul 28 17:24:03.787640 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:24:03.787614    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd4b27a_a718_4089_8a02_162bb1fda220.slice/crio-c98e6fd067a3a0081f77aabc885dcf3047313ca89fccf5a3b6a3600eba80e43e WatchSource:0}: Error finding container c98e6fd067a3a0081f77aabc885dcf3047313ca89fccf5a3b6a3600eba80e43e: Status 404 returned error can't find the container with id c98e6fd067a3a0081f77aabc885dcf3047313ca89fccf5a3b6a3600eba80e43e
Jul 28 17:24:03.803010 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:03.802988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vbjnd"]
Jul 28 17:24:03.806134 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:24:03.806105    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2143ac_9761_4e73_a1bb_38d7d84ce1b0.slice/crio-668ad7157bef05829ca8d2b1bfc4d6264d566c24a0d4f9ecec0cb6f4f9a8a7f2 WatchSource:0}: Error finding container 668ad7157bef05829ca8d2b1bfc4d6264d566c24a0d4f9ecec0cb6f4f9a8a7f2: Status 404 returned error can't find the container with id 668ad7157bef05829ca8d2b1bfc4d6264d566c24a0d4f9ecec0cb6f4f9a8a7f2
Jul 28 17:24:04.798190 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:04.798099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z7wgb" event={"ID":"8cd4b27a-a718-4089-8a02-162bb1fda220","Type":"ContainerStarted","Data":"c98e6fd067a3a0081f77aabc885dcf3047313ca89fccf5a3b6a3600eba80e43e"}
Jul 28 17:24:04.800808 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:04.800745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd" event={"ID":"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0","Type":"ContainerStarted","Data":"668ad7157bef05829ca8d2b1bfc4d6264d566c24a0d4f9ecec0cb6f4f9a8a7f2"}
Jul 28 17:24:07.816460 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.816423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd" event={"ID":"cc2143ac-9761-4e73-a1bb-38d7d84ce1b0","Type":"ContainerStarted","Data":"743d555f0aa43142833e17054f647159ff889af2baf2d0eecadeffa27c7a6588"}
Jul 28 17:24:07.816936 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.816549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:07.817919 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.817892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z7wgb" event={"ID":"8cd4b27a-a718-4089-8a02-162bb1fda220","Type":"ContainerStarted","Data":"4d5546d205ee775527b6ae6776f61527474dcb46e9e93684288cd9738b924c1f"}
Jul 28 17:24:07.818045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.818034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:07.835559 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.835451    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd" podStartSLOduration=1.192995139 podStartE2EDuration="4.835434336s" totalImagesPullingTime="3.642439197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:07.834922144 +0000 UTC m=+416.152361033" watchObservedRunningTime="2026-07-28 17:24:07.835434336 +0000 UTC m=+416.152873225"
Jul 28 17:24:07.853204 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:07.853153    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-z7wgb" podStartSLOduration=1.237087408 podStartE2EDuration="4.853137993s" totalImagesPullingTime="3.616050585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:07.852933065 +0000 UTC m=+416.170371951" watchObservedRunningTime="2026-07-28 17:24:07.853137993 +0000 UTC m=+416.170576882"
Jul 28 17:24:17.829051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:17.828997    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679b96ff76-djzpb" podUID="b364c70c-526b-4734-9671-18f59649456a" containerName="console" containerID="cri-o://5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3" gracePeriod=15
Jul 28 17:24:18.076550 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.076525    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:24:18.218475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218446    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218651 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218487    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218651 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218505    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218651 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218709    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r26jj\" (UniqueName: \"kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218819    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert\") pod \"b364c70c-526b-4734-9671-18f59649456a\" (UID: \"b364c70c-526b-4734-9671-18f59649456a\") "
Jul 28 17:24:18.218975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218924    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca" (OuterVolumeSpecName: "service-ca") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:18.218975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:18.219063 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.218990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config" (OuterVolumeSpecName: "console-config") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:18.219209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.219187    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-trusted-ca-bundle\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.219209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.219211    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-service-ca\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.219404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.219223    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-console-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.219404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.219241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:18.220667 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.220637    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:18.220754 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.220718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:18.220754 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.220725    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj" (OuterVolumeSpecName: "kube-api-access-r26jj") pod "b364c70c-526b-4734-9671-18f59649456a" (UID: "b364c70c-526b-4734-9671-18f59649456a"). InnerVolumeSpecName "kube-api-access-r26jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:18.320403 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.320371    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b364c70c-526b-4734-9671-18f59649456a-oauth-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.320403 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.320396    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-serving-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.320403 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.320405    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b364c70c-526b-4734-9671-18f59649456a-console-oauth-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.320677 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.320414    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26jj\" (UniqueName: \"kubernetes.io/projected/b364c70c-526b-4734-9671-18f59649456a-kube-api-access-r26jj\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:24:18.824133 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.824103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-vbjnd"
Jul 28 17:24:18.826183 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.826157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-z7wgb"
Jul 28 17:24:18.861828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.861800    2561 generic.go:356] "Generic (PLEG): container finished" podID="b364c70c-526b-4734-9671-18f59649456a" containerID="5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3" exitCode=0
Jul 28 17:24:18.862228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.861857    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b96ff76-djzpb"
Jul 28 17:24:18.862228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.861887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b96ff76-djzpb" event={"ID":"b364c70c-526b-4734-9671-18f59649456a","Type":"ContainerDied","Data":"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"}
Jul 28 17:24:18.862228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.861944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b96ff76-djzpb" event={"ID":"b364c70c-526b-4734-9671-18f59649456a","Type":"ContainerDied","Data":"9f59a994abfcd7c71ad976efa2db4b4e598b4f1fd069db00f4e7875e0d618d8b"}
Jul 28 17:24:18.862228 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.861971    2561 scope.go:117] "RemoveContainer" containerID="5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"
Jul 28 17:24:18.871913 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.871894    2561 scope.go:117] "RemoveContainer" containerID="5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"
Jul 28 17:24:18.872228 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:24:18.872205    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3\": container with ID starting with 5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3 not found: ID does not exist" containerID="5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"
Jul 28 17:24:18.872318 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.872241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3"} err="failed to get container status \"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3\": rpc error: code = NotFound desc = could not find container \"5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3\": container with ID starting with 5059c430b49272f3eb8e072de4894ac1cb051a03d69b208daddca2d49dd205d3 not found: ID does not exist"
Jul 28 17:24:18.889959 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.889931    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:24:18.893164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:18.893141    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679b96ff76-djzpb"]
Jul 28 17:24:19.673626 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.673593    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qxz4t"]
Jul 28 17:24:19.673890 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.673878    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b364c70c-526b-4734-9671-18f59649456a" containerName="console"
Jul 28 17:24:19.673933 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.673891    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b364c70c-526b-4734-9671-18f59649456a" containerName="console"
Jul 28 17:24:19.673966 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.673936    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b364c70c-526b-4734-9671-18f59649456a" containerName="console"
Jul 28 17:24:19.678216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.678201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.684952 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.684749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qxz4t"]
Jul 28 17:24:19.732293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.732265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvjm\" (UniqueName: \"kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.732446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.732301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.833102 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.833073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvjm\" (UniqueName: \"kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.833277 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.833116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.833492 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.833473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.842548 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.842523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvjm\" (UniqueName: \"kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm\") pod \"s3-init-qxz4t\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") " pod="kserve/s3-init-qxz4t"
Jul 28 17:24:19.997708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:19.997628    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxz4t"
Jul 28 17:24:20.130804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:20.130766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qxz4t"]
Jul 28 17:24:20.134012 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:24:20.133983    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80322afd_413f_4a62_b09b_e0f60938ae54.slice/crio-1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00 WatchSource:0}: Error finding container 1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00: Status 404 returned error can't find the container with id 1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00
Jul 28 17:24:20.202688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:20.202652    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b364c70c-526b-4734-9671-18f59649456a" path="/var/lib/kubelet/pods/b364c70c-526b-4734-9671-18f59649456a/volumes"
Jul 28 17:24:20.870995 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:20.870960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerStarted","Data":"1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00"}
Jul 28 17:24:24.890812 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:24.890730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerStarted","Data":"c78ef662cd447959d617f61ba85f277ef3b7f2c90955ef32b7f7809192b242b9"}
Jul 28 17:24:31.918695 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:31.918658    2561 generic.go:356] "Generic (PLEG): container finished" podID="80322afd-413f-4a62-b09b-e0f60938ae54" containerID="c78ef662cd447959d617f61ba85f277ef3b7f2c90955ef32b7f7809192b242b9" exitCode=0
Jul 28 17:24:31.919054 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:31.918736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerDied","Data":"c78ef662cd447959d617f61ba85f277ef3b7f2c90955ef32b7f7809192b242b9"}
Jul 28 17:24:36.945889 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:36.945857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerStarted","Data":"9ca2a60f6ccb6b17ad5c718e5a42c80f07546747325bb110ac4e43c1d9fcb635"}
Jul 28 17:24:36.964873 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:24:36.964825    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qxz4t" podStartSLOduration=1.950593908 podStartE2EDuration="17.964811144s" totalImagesPullingTime="8.652673127s" totalInitContainerRuntime="7.361544109s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:36.962522649 +0000 UTC m=+445.279961536" watchObservedRunningTime="2026-07-28 17:24:36.964811144 +0000 UTC m=+445.282250032"
Jul 28 17:25:42.181471 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:42.181435    2561 generic.go:356] "Generic (PLEG): container finished" podID="80322afd-413f-4a62-b09b-e0f60938ae54" containerID="9ca2a60f6ccb6b17ad5c718e5a42c80f07546747325bb110ac4e43c1d9fcb635" exitCode=0
Jul 28 17:25:42.182047 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:42.181503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerDied","Data":"9ca2a60f6ccb6b17ad5c718e5a42c80f07546747325bb110ac4e43c1d9fcb635"}
Jul 28 17:25:43.311493 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.311466    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxz4t"
Jul 28 17:25:43.503199 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.503117    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpvjm\" (UniqueName: \"kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm\") pod \"80322afd-413f-4a62-b09b-e0f60938ae54\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") "
Jul 28 17:25:43.503380 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.503204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache\") pod \"80322afd-413f-4a62-b09b-e0f60938ae54\" (UID: \"80322afd-413f-4a62-b09b-e0f60938ae54\") "
Jul 28 17:25:43.505256 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.505228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm" (OuterVolumeSpecName: "kube-api-access-tpvjm") pod "80322afd-413f-4a62-b09b-e0f60938ae54" (UID: "80322afd-413f-4a62-b09b-e0f60938ae54"). InnerVolumeSpecName "kube-api-access-tpvjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:43.527180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.527145    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache" (OuterVolumeSpecName: "model-cache") pod "80322afd-413f-4a62-b09b-e0f60938ae54" (UID: "80322afd-413f-4a62-b09b-e0f60938ae54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:43.604211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.604181    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvjm\" (UniqueName: \"kubernetes.io/projected/80322afd-413f-4a62-b09b-e0f60938ae54-kube-api-access-tpvjm\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:25:43.604211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:43.604208    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80322afd-413f-4a62-b09b-e0f60938ae54-model-cache\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:25:44.190541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.190519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxz4t"
Jul 28 17:25:44.190541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.190525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxz4t" event={"ID":"80322afd-413f-4a62-b09b-e0f60938ae54","Type":"ContainerDied","Data":"1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00"}
Jul 28 17:25:44.190752 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.190571    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f73785a49166c7fd32edf47f046c9f738140f950e774b4931e1acd3c7a3fe00"
Jul 28 17:25:44.971771 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.971741    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:44.972164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.972053    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80322afd-413f-4a62-b09b-e0f60938ae54" containerName="s3-init"
Jul 28 17:25:44.972164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.972064    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80322afd-413f-4a62-b09b-e0f60938ae54" containerName="s3-init"
Jul 28 17:25:44.972164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.972088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80322afd-413f-4a62-b09b-e0f60938ae54" containerName="download-hf-model"
Jul 28 17:25:44.972164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.972096    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80322afd-413f-4a62-b09b-e0f60938ae54" containerName="download-hf-model"
Jul 28 17:25:44.972164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.972145    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="80322afd-413f-4a62-b09b-e0f60938ae54" containerName="s3-init"
Jul 28 17:25:44.974914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.974899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:44.977598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.977579    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 17:25:44.983325 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:44.983283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:45.117985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.117947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.117985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.117992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2wn\" (UniqueName: \"kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.218784 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.218752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.218784 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.218791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2wn\" (UniqueName: \"kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.219126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.219108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.229016 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.228938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2wn\" (UniqueName: \"kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn\") pod \"seaweedfs-tls-custom-6644bf5d6c-wjkkd\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.283812 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.283774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:45.410841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:45.410766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:45.413516 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:25:45.413486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e662728_21a6_4fca_a419_183fbd32a1c4.slice/crio-1147c02393f7ae5606e77d54436a12cf3b5ba9d1256f0365be9a0c34fa3a1533 WatchSource:0}: Error finding container 1147c02393f7ae5606e77d54436a12cf3b5ba9d1256f0365be9a0c34fa3a1533: Status 404 returned error can't find the container with id 1147c02393f7ae5606e77d54436a12cf3b5ba9d1256f0365be9a0c34fa3a1533
Jul 28 17:25:46.203762 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:46.203729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" event={"ID":"6e662728-21a6-4fca-a419-183fbd32a1c4","Type":"ContainerStarted","Data":"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"}
Jul 28 17:25:46.204125 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:46.203778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" event={"ID":"6e662728-21a6-4fca-a419-183fbd32a1c4","Type":"ContainerStarted","Data":"1147c02393f7ae5606e77d54436a12cf3b5ba9d1256f0365be9a0c34fa3a1533"}
Jul 28 17:25:46.216620 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:46.216568    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" podStartSLOduration=1.954136997 podStartE2EDuration="2.216552042s" totalImagesPullingTime="262.415045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:46.215096149 +0000 UTC m=+514.532535036" watchObservedRunningTime="2026-07-28 17:25:46.216552042 +0000 UTC m=+514.533990929"
Jul 28 17:25:47.765069 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:47.765033    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:48.206815 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:48.206776    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" podUID="6e662728-21a6-4fca-a419-183fbd32a1c4" containerName="seaweedfs-tls-custom" containerID="cri-o://5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e" gracePeriod=30
Jul 28 17:25:49.450204 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.450183    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:49.548475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.548404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c2wn\" (UniqueName: \"kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn\") pod \"6e662728-21a6-4fca-a419-183fbd32a1c4\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") "
Jul 28 17:25:49.548475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.548470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data\") pod \"6e662728-21a6-4fca-a419-183fbd32a1c4\" (UID: \"6e662728-21a6-4fca-a419-183fbd32a1c4\") "
Jul 28 17:25:49.549737 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.549714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data" (OuterVolumeSpecName: "data") pod "6e662728-21a6-4fca-a419-183fbd32a1c4" (UID: "6e662728-21a6-4fca-a419-183fbd32a1c4"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:49.550519 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.550498    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn" (OuterVolumeSpecName: "kube-api-access-2c2wn") pod "6e662728-21a6-4fca-a419-183fbd32a1c4" (UID: "6e662728-21a6-4fca-a419-183fbd32a1c4"). InnerVolumeSpecName "kube-api-access-2c2wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:49.649521 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.649492    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2wn\" (UniqueName: \"kubernetes.io/projected/6e662728-21a6-4fca-a419-183fbd32a1c4-kube-api-access-2c2wn\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:25:49.649521 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:49.649517    2561 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6e662728-21a6-4fca-a419-183fbd32a1c4-data\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:25:50.214690 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.214657    2561 generic.go:356] "Generic (PLEG): container finished" podID="6e662728-21a6-4fca-a419-183fbd32a1c4" containerID="5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e" exitCode=0
Jul 28 17:25:50.214818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.214698    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"
Jul 28 17:25:50.214818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.214734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" event={"ID":"6e662728-21a6-4fca-a419-183fbd32a1c4","Type":"ContainerDied","Data":"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"}
Jul 28 17:25:50.214818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.214782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd" event={"ID":"6e662728-21a6-4fca-a419-183fbd32a1c4","Type":"ContainerDied","Data":"1147c02393f7ae5606e77d54436a12cf3b5ba9d1256f0365be9a0c34fa3a1533"}
Jul 28 17:25:50.214818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.214804    2561 scope.go:117] "RemoveContainer" containerID="5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"
Jul 28 17:25:50.223625 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.223611    2561 scope.go:117] "RemoveContainer" containerID="5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"
Jul 28 17:25:50.223862 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:25:50.223844    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e\": container with ID starting with 5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e not found: ID does not exist" containerID="5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"
Jul 28 17:25:50.223908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.223869    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e"} err="failed to get container status \"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e\": rpc error: code = NotFound desc = could not find container \"5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e\": container with ID starting with 5f75fef97b2f6c10de8c49f50589e23c263bcb7085f233278f4c432d507bcf9e not found: ID does not exist"
Jul 28 17:25:50.233365 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.233342    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:50.240109 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.240089    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-wjkkd"]
Jul 28 17:25:50.271567 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.271543    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"]
Jul 28 17:25:50.271820 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.271809    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e662728-21a6-4fca-a419-183fbd32a1c4" containerName="seaweedfs-tls-custom"
Jul 28 17:25:50.271860 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.271821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e662728-21a6-4fca-a419-183fbd32a1c4" containerName="seaweedfs-tls-custom"
Jul 28 17:25:50.271893 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.271874    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e662728-21a6-4fca-a419-183fbd32a1c4" containerName="seaweedfs-tls-custom"
Jul 28 17:25:50.276014 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.276000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.278863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.278844    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 17:25:50.278961 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.278874    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 28 17:25:50.289753 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.289730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"]
Jul 28 17:25:50.354749 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.354720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkx7\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-kube-api-access-qfkx7\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.354871 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.354760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.354871 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.354783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0509f459-d23e-4311-b7b9-805790d27b9b-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.455981 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.455946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkx7\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-kube-api-access-qfkx7\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.455981 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.455986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.456404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.456005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0509f459-d23e-4311-b7b9-805790d27b9b-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.456404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.456341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0509f459-d23e-4311-b7b9-805790d27b9b-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.458360 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.458337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.465136 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.465083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkx7\" (UniqueName: \"kubernetes.io/projected/0509f459-d23e-4311-b7b9-805790d27b9b-kube-api-access-qfkx7\") pod \"seaweedfs-tls-custom-8b44d767d-qqsn5\" (UID: \"0509f459-d23e-4311-b7b9-805790d27b9b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.585351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.585292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"
Jul 28 17:25:50.713553 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:50.713515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqsn5"]
Jul 28 17:25:50.721534 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:25:50.721510    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0509f459_d23e_4311_b7b9_805790d27b9b.slice/crio-ba79e6fde72a747836b839761d0b64b989e54d741ef25e045a9a0aa75ad0f8b0 WatchSource:0}: Error finding container ba79e6fde72a747836b839761d0b64b989e54d741ef25e045a9a0aa75ad0f8b0: Status 404 returned error can't find the container with id ba79e6fde72a747836b839761d0b64b989e54d741ef25e045a9a0aa75ad0f8b0
Jul 28 17:25:51.220163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.220129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5" event={"ID":"0509f459-d23e-4311-b7b9-805790d27b9b","Type":"ContainerStarted","Data":"65ec855c5779c50d4818d5aa1fe8c7ca08a4a09af04077aa1fe3b21a9d13cf75"}
Jul 28 17:25:51.220385 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.220183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5" event={"ID":"0509f459-d23e-4311-b7b9-805790d27b9b","Type":"ContainerStarted","Data":"ba79e6fde72a747836b839761d0b64b989e54d741ef25e045a9a0aa75ad0f8b0"}
Jul 28 17:25:51.236970 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.236887    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqsn5" podStartSLOduration=0.978700715 podStartE2EDuration="1.236870912s" totalImagesPullingTime="258.170197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:51.236369478 +0000 UTC m=+519.553808366" watchObservedRunningTime="2026-07-28 17:25:51.236870912 +0000 UTC m=+519.554309811"
Jul 28 17:25:51.565204 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.565133    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-c8prc"]
Jul 28 17:25:51.568251 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.568235    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.577752 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.577729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-c8prc"]
Jul 28 17:25:51.664825 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.664794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.664962 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.664836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5k5\" (UniqueName: \"kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.765200 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.765172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.765375 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.765215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5k5\" (UniqueName: \"kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.765535 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.765517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.774102 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.774084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5k5\" (UniqueName: \"kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5\") pod \"s3-tls-init-custom-c8prc\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") " pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:51.889802 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:51.889748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:25:52.018382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:52.018357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-c8prc"]
Jul 28 17:25:52.020640 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:25:52.020615    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18b17d9e_8e6d_49e2_8502_2300af7cf42c.slice/crio-cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289 WatchSource:0}: Error finding container cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289: Status 404 returned error can't find the container with id cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289
Jul 28 17:25:52.205905 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:52.205875    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e662728-21a6-4fca-a419-183fbd32a1c4" path="/var/lib/kubelet/pods/6e662728-21a6-4fca-a419-183fbd32a1c4/volumes"
Jul 28 17:25:52.226126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:52.226097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerStarted","Data":"01297a9e8b095ce8e72a85a8f3f6eceb65874d944e9b30a4d9afe5732e374ca1"}
Jul 28 17:25:52.226344 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:52.226137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerStarted","Data":"cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289"}
Jul 28 17:25:59.251757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:59.251676    2561 generic.go:356] "Generic (PLEG): container finished" podID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerID="01297a9e8b095ce8e72a85a8f3f6eceb65874d944e9b30a4d9afe5732e374ca1" exitCode=0
Jul 28 17:25:59.251757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:25:59.251733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerDied","Data":"01297a9e8b095ce8e72a85a8f3f6eceb65874d944e9b30a4d9afe5732e374ca1"}
Jul 28 17:26:00.257259 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:00.257222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerStarted","Data":"ab92c3fb02e25d17bbe3765a0e0352cb0b0fedec10dd892246a6b41173ebb600"}
Jul 28 17:26:00.275603 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:00.275553    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-c8prc" podStartSLOduration=2.38564169 podStartE2EDuration="9.275532901s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.889891211s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:00.274707151 +0000 UTC m=+528.592146039" watchObservedRunningTime="2026-07-28 17:26:00.275532901 +0000 UTC m=+528.592971788"
Jul 28 17:26:06.284467 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:06.284436    2561 generic.go:356] "Generic (PLEG): container finished" podID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerID="ab92c3fb02e25d17bbe3765a0e0352cb0b0fedec10dd892246a6b41173ebb600" exitCode=0
Jul 28 17:26:06.284841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:06.284503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerDied","Data":"ab92c3fb02e25d17bbe3765a0e0352cb0b0fedec10dd892246a6b41173ebb600"}
Jul 28 17:26:07.428045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.428024    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:26:07.501215 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.501189    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z5k5\" (UniqueName: \"kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5\") pod \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") "
Jul 28 17:26:07.501401 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.501223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache\") pod \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\" (UID: \"18b17d9e-8e6d-49e2-8502-2300af7cf42c\") "
Jul 28 17:26:07.503242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.503205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5" (OuterVolumeSpecName: "kube-api-access-4z5k5") pod "18b17d9e-8e6d-49e2-8502-2300af7cf42c" (UID: "18b17d9e-8e6d-49e2-8502-2300af7cf42c"). InnerVolumeSpecName "kube-api-access-4z5k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:07.519598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.519568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache" (OuterVolumeSpecName: "model-cache") pod "18b17d9e-8e6d-49e2-8502-2300af7cf42c" (UID: "18b17d9e-8e6d-49e2-8502-2300af7cf42c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:07.602119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.602039    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z5k5\" (UniqueName: \"kubernetes.io/projected/18b17d9e-8e6d-49e2-8502-2300af7cf42c-kube-api-access-4z5k5\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:26:07.602119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:07.602069    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b17d9e-8e6d-49e2-8502-2300af7cf42c-model-cache\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.292384 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:08.292347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-c8prc" event={"ID":"18b17d9e-8e6d-49e2-8502-2300af7cf42c","Type":"ContainerDied","Data":"cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289"}
Jul 28 17:26:08.292545 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:08.292392    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf2ce77f8b2f9c1dfe952ed0d52f4a8b7b0fa7c14c46afa1a48735ac1907289"
Jul 28 17:26:08.292545 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:08.292364    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-c8prc"
Jul 28 17:26:13.497619 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497580    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4s49f"]
Jul 28 17:26:13.498018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497860    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerName="s3-tls-init-custom"
Jul 28 17:26:13.498018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497871    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerName="s3-tls-init-custom"
Jul 28 17:26:13.498018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497888    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerName="download-hf-model"
Jul 28 17:26:13.498018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497894    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerName="download-hf-model"
Jul 28 17:26:13.498018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.497946    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b17d9e-8e6d-49e2-8502-2300af7cf42c" containerName="s3-tls-init-custom"
Jul 28 17:26:13.502861 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.502844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.505435 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.505416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 17:26:13.510443 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.510423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4s49f"]
Jul 28 17:26:13.651175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.651140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.651378 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.651183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjkx\" (UniqueName: \"kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.752570 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.752484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.752570 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.752545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjkx\" (UniqueName: \"kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.752955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.752932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.761882 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.761856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjkx\" (UniqueName: \"kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx\") pod \"s3-tls-init-serving-4s49f\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") " pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.823927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.823894    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:13.949686 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:13.949607    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4s49f"]
Jul 28 17:26:13.952619 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:26:13.952590    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a8a83f5_a79d_4594_8ee0_e2c7ea338977.slice/crio-bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76 WatchSource:0}: Error finding container bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76: Status 404 returned error can't find the container with id bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76
Jul 28 17:26:14.317589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:14.317502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerStarted","Data":"5be7d73c0de48411c49d75564b04ca9b72eae91c622865d1b073078ab41079bc"}
Jul 28 17:26:14.317589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:14.317546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerStarted","Data":"bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76"}
Jul 28 17:26:20.342508 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:20.342463    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerID="5be7d73c0de48411c49d75564b04ca9b72eae91c622865d1b073078ab41079bc" exitCode=0
Jul 28 17:26:20.342914 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:20.342538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerDied","Data":"5be7d73c0de48411c49d75564b04ca9b72eae91c622865d1b073078ab41079bc"}
Jul 28 17:26:21.348183 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:21.348151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerStarted","Data":"06e71f974083fd84991594af791b08bc0a134b7b6cd12c5e60594e026927824f"}
Jul 28 17:26:21.368807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:21.368754    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4s49f" podStartSLOduration=2.208907703 podStartE2EDuration="8.368738863s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.15983116s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:21.367563416 +0000 UTC m=+549.685002321" watchObservedRunningTime="2026-07-28 17:26:21.368738863 +0000 UTC m=+549.686177772"
Jul 28 17:26:28.379503 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:28.379467    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerID="06e71f974083fd84991594af791b08bc0a134b7b6cd12c5e60594e026927824f" exitCode=0
Jul 28 17:26:28.379881 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:28.379537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerDied","Data":"06e71f974083fd84991594af791b08bc0a134b7b6cd12c5e60594e026927824f"}
Jul 28 17:26:29.526372 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.526346    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:29.695673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.695645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache\") pod \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") "
Jul 28 17:26:29.695853 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.695730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bjkx\" (UniqueName: \"kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx\") pod \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\" (UID: \"7a8a83f5-a79d-4594-8ee0-e2c7ea338977\") "
Jul 28 17:26:29.697838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.697798    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx" (OuterVolumeSpecName: "kube-api-access-7bjkx") pod "7a8a83f5-a79d-4594-8ee0-e2c7ea338977" (UID: "7a8a83f5-a79d-4594-8ee0-e2c7ea338977"). InnerVolumeSpecName "kube-api-access-7bjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:29.715227 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.715198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a8a83f5-a79d-4594-8ee0-e2c7ea338977" (UID: "7a8a83f5-a79d-4594-8ee0-e2c7ea338977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:29.796675 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.796644    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-model-cache\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:26:29.796675 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:29.796671    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjkx\" (UniqueName: \"kubernetes.io/projected/7a8a83f5-a79d-4594-8ee0-e2c7ea338977-kube-api-access-7bjkx\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.388041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:30.388010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4s49f" event={"ID":"7a8a83f5-a79d-4594-8ee0-e2c7ea338977","Type":"ContainerDied","Data":"bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76"}
Jul 28 17:26:30.388041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:30.388030    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4s49f"
Jul 28 17:26:30.388041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:30.388047    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad7a01df27c1531f8674c4aefa46fd1c131e643aebc5900f38aadb6b84bab76"
Jul 28 17:26:42.719790 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.719753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:26:42.720182 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.720054    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerName="download-hf-model"
Jul 28 17:26:42.720182 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.720064    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerName="download-hf-model"
Jul 28 17:26:42.720182 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.720073    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerName="s3-tls-init-serving"
Jul 28 17:26:42.720182 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.720079    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerName="s3-tls-init-serving"
Jul 28 17:26:42.720182 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.720137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8a83f5-a79d-4594-8ee0-e2c7ea338977" containerName="s3-tls-init-serving"
Jul 28 17:26:42.723256 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.723236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.725924 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.725899    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:26:42.726222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.726200    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:26:42.726843 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.726815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:26:42.726843 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.726828    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:26:42.726980 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.726884    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 17:26:42.726980 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.726947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:26:42.727075 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.727063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 17:26:42.733852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.733829    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:26:42.808160 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.808126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.808160 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.808165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.808412 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.808196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvt6\" (UniqueName: \"kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.808412 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.808349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.808412 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.808402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909454 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvt6\" (UniqueName: \"kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.909913 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.909887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.910296 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.910266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.910365 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.910266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.911971 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.911950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:42.918078 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:42.918052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvt6\" (UniqueName: \"kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:43.034264 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:43.034186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:26:43.165819 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:43.165788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:26:43.169023 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:26:43.169000    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c48de3_1844_4d33_bacd_455efa0aca8c.slice/crio-edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254 WatchSource:0}: Error finding container edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254: Status 404 returned error can't find the container with id edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254
Jul 28 17:26:43.457339 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:43.457289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerStarted","Data":"31283b399eca3ae638caf63efec5b0ebdb96c1160bdc188b0ceca1cd03b262d7"}
Jul 28 17:26:43.457543 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:43.457392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerStarted","Data":"edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254"}
Jul 28 17:26:47.474651 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:47.474617    2561 generic.go:356] "Generic (PLEG): container finished" podID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerID="31283b399eca3ae638caf63efec5b0ebdb96c1160bdc188b0ceca1cd03b262d7" exitCode=0
Jul 28 17:26:47.475027 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:26:47.474682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerDied","Data":"31283b399eca3ae638caf63efec5b0ebdb96c1160bdc188b0ceca1cd03b262d7"}
Jul 28 17:27:01.556813 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:01.556752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerStarted","Data":"82011f82291d211835ed9e4564da6bfeb13defff619725f396ca6efc3b359c85"}
Jul 28 17:27:03.567702 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:03.567670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerStarted","Data":"bb25f662dc2cab375073535187f660889971aafabe62bd9f099b6484d3aefa89"}
Jul 28 17:27:06.581455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:06.581413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerStarted","Data":"06d36d07a58e1f7021989816d17216282800c1da693468b432028d09beb1fcd8"}
Jul 28 17:27:06.581828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:06.581584    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:27:06.603976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:06.603901    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podStartSLOduration=2.506775656 podStartE2EDuration="24.603887219s" totalImagesPullingTime="18.481631592s" totalInitContainerRuntime="3.615479971s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:06.602350158 +0000 UTC m=+594.919789046" watchObservedRunningTime="2026-07-28 17:27:06.603887219 +0000 UTC m=+594.921326105"
Jul 28 17:27:07.586119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:07.586089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:27:07.586522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:07.586167    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:27:07.587561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:07.587531    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:07.588279 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:07.588255    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:08.591222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:08.591166    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:08.591731 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:08.591671    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:08.595025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:08.595007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:27:09.595454 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:09.595414    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:09.595872 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:09.595635    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:19.596103 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:19.596061    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:19.596557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:19.596535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:29.595697 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:29.595655    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:29.596169 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:29.596090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:39.596387 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:39.596339    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:39.596911 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:39.596824    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:49.595372 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:49.595323    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:49.596638 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:49.595817    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:59.596096 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:59.596050    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:27:59.596563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:27:59.596534    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:09.595491 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:09.595450    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:28:09.595937 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:09.595914    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:19.596504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:19.596475    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:19.596978 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:19.596570    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:27.895242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.895207    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:28:27.895725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.895500    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" containerID="cri-o://82011f82291d211835ed9e4564da6bfeb13defff619725f396ca6efc3b359c85" gracePeriod=30
Jul 28 17:28:27.895725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.895529    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" containerID="cri-o://06d36d07a58e1f7021989816d17216282800c1da693468b432028d09beb1fcd8" gracePeriod=30
Jul 28 17:28:27.895725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.895541    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" containerID="cri-o://bb25f662dc2cab375073535187f660889971aafabe62bd9f099b6484d3aefa89" gracePeriod=30
Jul 28 17:28:27.917166 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.917143    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:28:27.920014 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.919995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:27.922344 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.922325    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 17:28:27.922503 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.922486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 17:28:27.933852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:27.933830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:28:28.012456 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.012427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.012607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.012466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.012607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.012483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpk9\" (UniqueName: \"kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.012607 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.012540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.012728 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.012622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.112993 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.112950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpk9\" (UniqueName: \"kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113443 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113734 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.113814 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.113746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.115495 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.115480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.122857 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.122837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpk9\" (UniqueName: \"kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.230116 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.230081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:28.357456 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.357434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:28:28.359719 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:28:28.359690    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce725fe_9f84_48cf_ac88_8429ce338c79.slice/crio-5583c40fb0c5fcc98eb11cf8ce556d5e8266a40f2f45e2c5f6313acf99c64046 WatchSource:0}: Error finding container 5583c40fb0c5fcc98eb11cf8ce556d5e8266a40f2f45e2c5f6313acf99c64046: Status 404 returned error can't find the container with id 5583c40fb0c5fcc98eb11cf8ce556d5e8266a40f2f45e2c5f6313acf99c64046
Jul 28 17:28:28.591971 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.591877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:28.909821 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.909793    2561 generic.go:356] "Generic (PLEG): container finished" podID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerID="bb25f662dc2cab375073535187f660889971aafabe62bd9f099b6484d3aefa89" exitCode=2
Jul 28 17:28:28.910244 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.909844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerDied","Data":"bb25f662dc2cab375073535187f660889971aafabe62bd9f099b6484d3aefa89"}
Jul 28 17:28:28.911385 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.911362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerStarted","Data":"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"}
Jul 28 17:28:28.911473 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:28.911424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerStarted","Data":"5583c40fb0c5fcc98eb11cf8ce556d5e8266a40f2f45e2c5f6313acf99c64046"}
Jul 28 17:28:29.595855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:29.595805    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:28:29.596122 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:29.596094    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:31.927252 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:31.927217    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerID="22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1" exitCode=0
Jul 28 17:28:31.927714 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:31.927284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerDied","Data":"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"}
Jul 28 17:28:32.934153 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.934124    2561 generic.go:356] "Generic (PLEG): container finished" podID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerID="82011f82291d211835ed9e4564da6bfeb13defff619725f396ca6efc3b359c85" exitCode=0
Jul 28 17:28:32.934558 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.934199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerDied","Data":"82011f82291d211835ed9e4564da6bfeb13defff619725f396ca6efc3b359c85"}
Jul 28 17:28:32.936356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.936334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerStarted","Data":"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"}
Jul 28 17:28:32.936462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.936395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerStarted","Data":"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"}
Jul 28 17:28:32.936462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.936421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerStarted","Data":"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"}
Jul 28 17:28:32.936652 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.936630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:32.936776 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.936675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:32.938160 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.938137    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:32.962706 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:32.962659    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podStartSLOduration=2.518812333 podStartE2EDuration="5.962643627s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.443831294s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:32.960185877 +0000 UTC m=+681.277624798" watchObservedRunningTime="2026-07-28 17:28:32.962643627 +0000 UTC m=+681.280082512"
Jul 28 17:28:33.591524 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:33.591484    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:33.940116 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:33.940083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:33.940526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:33.940205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:33.941331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:33.941286    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:34.943877 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:34.943841    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:34.944361 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:34.944342    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:38.592177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:38.592134    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:38.592589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:38.592253    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:39.595374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:39.595328    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:28:39.595808 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:39.595637    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:39.948352 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:39.948327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:28:39.948934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:39.948901    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:39.949160 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:39.949131    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:43.591873 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:43.591832    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:48.592449 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:48.592400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:49.595631 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.595588    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:28:49.596020 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.595757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:49.596020 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.595896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:49.596020 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.596011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:49.949340 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.949283    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:49.949709 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:49.949688    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:53.591675 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:53.591630    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:28:58.035424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.035392    2561 generic.go:356] "Generic (PLEG): container finished" podID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerID="06d36d07a58e1f7021989816d17216282800c1da693468b432028d09beb1fcd8" exitCode=0
Jul 28 17:28:58.035729 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.035461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerDied","Data":"06d36d07a58e1f7021989816d17216282800c1da693468b432028d09beb1fcd8"}
Jul 28 17:28:58.035729 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.035546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb" event={"ID":"60c48de3-1844-4d33-bacd-455efa0aca8c","Type":"ContainerDied","Data":"edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254"}
Jul 28 17:28:58.035729 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.035571    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf6c026b6d601a08932725a9d86818cec3359f120608f918a23841396c62254"
Jul 28 17:28:58.039554 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.039534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:58.064060 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert\") pod \"60c48de3-1844-4d33-bacd-455efa0aca8c\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") "
Jul 28 17:28:58.064210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location\") pod \"60c48de3-1844-4d33-bacd-455efa0aca8c\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") "
Jul 28 17:28:58.064210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls\") pod \"60c48de3-1844-4d33-bacd-455efa0aca8c\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") "
Jul 28 17:28:58.064210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064134    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config\") pod \"60c48de3-1844-4d33-bacd-455efa0aca8c\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") "
Jul 28 17:28:58.064210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmvt6\" (UniqueName: \"kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6\") pod \"60c48de3-1844-4d33-bacd-455efa0aca8c\" (UID: \"60c48de3-1844-4d33-bacd-455efa0aca8c\") "
Jul 28 17:28:58.064520 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60c48de3-1844-4d33-bacd-455efa0aca8c" (UID: "60c48de3-1844-4d33-bacd-455efa0aca8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:58.064520 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60c48de3-1844-4d33-bacd-455efa0aca8c" (UID: "60c48de3-1844-4d33-bacd-455efa0aca8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:28:58.064703 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.064570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60c48de3-1844-4d33-bacd-455efa0aca8c" (UID: "60c48de3-1844-4d33-bacd-455efa0aca8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:58.066579 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.066552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6" (OuterVolumeSpecName: "kube-api-access-zmvt6") pod "60c48de3-1844-4d33-bacd-455efa0aca8c" (UID: "60c48de3-1844-4d33-bacd-455efa0aca8c"). InnerVolumeSpecName "kube-api-access-zmvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:28:58.066668 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.066610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60c48de3-1844-4d33-bacd-455efa0aca8c" (UID: "60c48de3-1844-4d33-bacd-455efa0aca8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:28:58.164841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.164811    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:28:58.164841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.164841    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvt6\" (UniqueName: \"kubernetes.io/projected/60c48de3-1844-4d33-bacd-455efa0aca8c-kube-api-access-zmvt6\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:28:58.165014 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.164850    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60c48de3-1844-4d33-bacd-455efa0aca8c-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:28:58.165014 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.164859    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60c48de3-1844-4d33-bacd-455efa0aca8c-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:28:58.165014 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:58.164868    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c48de3-1844-4d33-bacd-455efa0aca8c-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:28:59.039700 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:59.039672    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"
Jul 28 17:28:59.100610 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:59.100578    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:28:59.101949 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:59.101930    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb"]
Jul 28 17:28:59.949570 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:59.949522    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:28:59.950007 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:28:59.949985    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:00.203635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:00.203556    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" path="/var/lib/kubelet/pods/60c48de3-1844-4d33-bacd-455efa0aca8c/volumes"
Jul 28 17:29:09.949809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:09.949754    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:29:09.950201 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:09.950135    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:19.949042 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:19.948996    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:29:19.949469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:19.949448    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:29.948967 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:29.948923    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:29:29.949448 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:29.949409    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:39.948857 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:39.948814    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:29:39.949322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:39.949239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:49.949483 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:49.949454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:29:49.949965 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:29:49.949941    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:03.077721 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.077685    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078123    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078141    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078157    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="storage-initializer"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078166    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="storage-initializer"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078185    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078193    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078215    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy"
Jul 28 17:30:03.078271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078223    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy"
Jul 28 17:30:03.078742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kserve-container"
Jul 28 17:30:03.078742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078342    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="agent"
Jul 28 17:30:03.078742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.078353    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c48de3-1844-4d33-bacd-455efa0aca8c" containerName="kube-rbac-proxy"
Jul 28 17:30:03.080564 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.080543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.083212 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.083195    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 17:30:03.083355 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.083218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 17:30:03.092429 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.092405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:30:03.107276 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.107245    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:30:03.107637 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.107599    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" containerID="cri-o://c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42" gracePeriod=30
Jul 28 17:30:03.107773 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.107648    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" containerID="cri-o://64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6" gracePeriod=30
Jul 28 17:30:03.107773 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.107635    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" containerID="cri-o://234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6" gracePeriod=30
Jul 28 17:30:03.180957 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.180933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.181085 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.180984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqv8x\" (UniqueName: \"kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.181085 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.181038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.281594 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.281564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqv8x\" (UniqueName: \"kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.281734 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.281642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.281734 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.281689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.282242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.282221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.284025 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.283999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.289687 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.289665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqv8x\" (UniqueName: \"kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x\") pod \"message-dumper-predictor-6575bcfdfd-v67qd\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.307199 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.307174    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerID="234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6" exitCode=2
Jul 28 17:30:03.307339 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.307255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerDied","Data":"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"}
Jul 28 17:30:03.395476 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.395408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:03.521527 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.521505    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:30:03.523956 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:30:03.523930    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0203403d_801b_4f2f_89a1_1b221827bae4.slice/crio-89d025baeaced2eb94e35f0874f80bc471e8201b90690b1c65cef19fa94f8cea WatchSource:0}: Error finding container 89d025baeaced2eb94e35f0874f80bc471e8201b90690b1c65cef19fa94f8cea: Status 404 returned error can't find the container with id 89d025baeaced2eb94e35f0874f80bc471e8201b90690b1c65cef19fa94f8cea
Jul 28 17:30:03.525735 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:03.525719    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:30:04.312522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:04.312492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerStarted","Data":"89d025baeaced2eb94e35f0874f80bc471e8201b90690b1c65cef19fa94f8cea"}
Jul 28 17:30:04.945041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:04.945000    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:05.318300 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:05.318258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerStarted","Data":"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"}
Jul 28 17:30:05.318686 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:05.318334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerStarted","Data":"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"}
Jul 28 17:30:05.318686 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:05.318360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:05.338200 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:05.338145    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" podStartSLOduration=1.090047411 podStartE2EDuration="2.338127792s" totalImagesPullingTime="1.248080381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:05.337880684 +0000 UTC m=+773.655319570" watchObservedRunningTime="2026-07-28 17:30:05.338127792 +0000 UTC m=+773.655566678"
Jul 28 17:30:06.322734 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:06.322705    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:06.324379 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:06.324357    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:08.334140 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:08.334112    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerID="c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42" exitCode=0
Jul 28 17:30:08.334621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:08.334182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerDied","Data":"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"}
Jul 28 17:30:09.944528 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:09.944491    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:09.949833 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:09.949812    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:30:09.951648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:09.951610    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:13.338402 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:13.338374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:30:14.944515 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:14.944476    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:14.944900 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:14.944623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:19.944139 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:19.944096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:19.949502 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:19.949478    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:30:19.950992 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:19.950972    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:23.147660 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.147628    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:30:23.151232 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.151213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.155012 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.154990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 17:30:23.156545 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.156522    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 17:30:23.167918 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.167899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:30:23.250779 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.250749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.250934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.250809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwktr\" (UniqueName: \"kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.250934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.250829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.250934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.250853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.250934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.250879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.352417 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.352384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.352668 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.352650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.352849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.352834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwktr\" (UniqueName: \"kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.352956 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.352943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.353068 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.353055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.353901 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.353876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.354377 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.354353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.354628 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:23.354602    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 28 17:30:23.354728 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:23.354716    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls podName:d2f33d36-a65b-42ba-a4c3-bdce22a26efa nodeName:}" failed. No retries permitted until 2026-07-28 17:30:23.854684225 +0000 UTC m=+792.172123103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls") pod "isvc-logger-predictor-6b6746bb7-9l9d4" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa") : secret "isvc-logger-predictor-serving-cert" not found
Jul 28 17:30:23.354961 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.354931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.363243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.363219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwktr\" (UniqueName: \"kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.857623 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.857588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:23.859841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:23.859817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-9l9d4\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:24.060803 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:24.060761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:24.196011 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:24.195987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:30:24.197953 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:30:24.197932    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f33d36_a65b_42ba_a4c3_bdce22a26efa.slice/crio-7b722f23bcb01f02eba219845297eff1a78e2efe7826bafb39e15946609548be WatchSource:0}: Error finding container 7b722f23bcb01f02eba219845297eff1a78e2efe7826bafb39e15946609548be: Status 404 returned error can't find the container with id 7b722f23bcb01f02eba219845297eff1a78e2efe7826bafb39e15946609548be
Jul 28 17:30:24.400739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:24.400664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerStarted","Data":"42379c282c51916d03f5c5d902958cbca87a7c2f0a38c4a9ea8d53f62c006412"}
Jul 28 17:30:24.400739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:24.400729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerStarted","Data":"7b722f23bcb01f02eba219845297eff1a78e2efe7826bafb39e15946609548be"}
Jul 28 17:30:24.943993 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:24.943954    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:28.419655 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:28.419619    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerID="42379c282c51916d03f5c5d902958cbca87a7c2f0a38c4a9ea8d53f62c006412" exitCode=0
Jul 28 17:30:28.420127 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:28.419660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerDied","Data":"42379c282c51916d03f5c5d902958cbca87a7c2f0a38c4a9ea8d53f62c006412"}
Jul 28 17:30:29.427092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerStarted","Data":"518f892cf0e6320186f997d66550d8104a0a602a11b11746512d629f485925c8"}
Jul 28 17:30:29.427541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerStarted","Data":"b7678fcf982760ababdfb7bfb4c69d1c7d5d77c7943b2ebe88b5aba746af322b"}
Jul 28 17:30:29.427541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerStarted","Data":"d3218608b39c9cf9596cd26a07bf15aece4c42e51281c23f802242e0216be9f6"}
Jul 28 17:30:29.427541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:29.427708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:29.427708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.427595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:29.429069 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.429036    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:30:29.429684 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.429664    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:29.451035 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.450988    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podStartSLOduration=2.847300786 podStartE2EDuration="6.45097541s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.603674624s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:29.450294035 +0000 UTC m=+797.767732919" watchObservedRunningTime="2026-07-28 17:30:29.45097541 +0000 UTC m=+797.768414295"
Jul 28 17:30:29.944136 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.944097    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:30:29.949823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.949793    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 28 17:30:29.949912 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.949898    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:29.950442 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.950417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:29.950550 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:29.950505    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:30.432356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:30.432317    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:30:30.432801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:30.432778    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:33.254593 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.254568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:33.436459 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config\") pod \"9ce725fe-9f84-48cf-ac88-8429ce338c79\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") "
Jul 28 17:30:33.436646 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436472    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert\") pod \"9ce725fe-9f84-48cf-ac88-8429ce338c79\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") "
Jul 28 17:30:33.436646 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls\") pod \"9ce725fe-9f84-48cf-ac88-8429ce338c79\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") "
Jul 28 17:30:33.436759 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location\") pod \"9ce725fe-9f84-48cf-ac88-8429ce338c79\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") "
Jul 28 17:30:33.436759 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzpk9\" (UniqueName: \"kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9\") pod \"9ce725fe-9f84-48cf-ac88-8429ce338c79\" (UID: \"9ce725fe-9f84-48cf-ac88-8429ce338c79\") "
Jul 28 17:30:33.436858 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ce725fe-9f84-48cf-ac88-8429ce338c79" (UID: "9ce725fe-9f84-48cf-ac88-8429ce338c79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:33.436858 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ce725fe-9f84-48cf-ac88-8429ce338c79" (UID: "9ce725fe-9f84-48cf-ac88-8429ce338c79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:33.437006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.436983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ce725fe-9f84-48cf-ac88-8429ce338c79" (UID: "9ce725fe-9f84-48cf-ac88-8429ce338c79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:33.437126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.437108    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:30:33.437177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.437134    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ce725fe-9f84-48cf-ac88-8429ce338c79-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:30:33.437177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.437148    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ce725fe-9f84-48cf-ac88-8429ce338c79-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:30:33.438507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.438488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ce725fe-9f84-48cf-ac88-8429ce338c79" (UID: "9ce725fe-9f84-48cf-ac88-8429ce338c79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:33.438598 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.438574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9" (OuterVolumeSpecName: "kube-api-access-nzpk9") pod "9ce725fe-9f84-48cf-ac88-8429ce338c79" (UID: "9ce725fe-9f84-48cf-ac88-8429ce338c79"). InnerVolumeSpecName "kube-api-access-nzpk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:33.447434 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.447396    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerID="64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6" exitCode=0
Jul 28 17:30:33.447565 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.447473    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"
Jul 28 17:30:33.447565 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.447480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerDied","Data":"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"}
Jul 28 17:30:33.447676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.447569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8" event={"ID":"9ce725fe-9f84-48cf-ac88-8429ce338c79","Type":"ContainerDied","Data":"5583c40fb0c5fcc98eb11cf8ce556d5e8266a40f2f45e2c5f6313acf99c64046"}
Jul 28 17:30:33.447676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.447614    2561 scope.go:117] "RemoveContainer" containerID="64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"
Jul 28 17:30:33.455997 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.455911    2561 scope.go:117] "RemoveContainer" containerID="234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"
Jul 28 17:30:33.463299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.463276    2561 scope.go:117] "RemoveContainer" containerID="c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"
Jul 28 17:30:33.469619 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.469600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:30:33.470969 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.470901    2561 scope.go:117] "RemoveContainer" containerID="22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"
Jul 28 17:30:33.473289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.473272    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8"]
Jul 28 17:30:33.478142 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478129    2561 scope.go:117] "RemoveContainer" containerID="64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"
Jul 28 17:30:33.478433 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:33.478408    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6\": container with ID starting with 64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6 not found: ID does not exist" containerID="64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"
Jul 28 17:30:33.478505 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478440    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6"} err="failed to get container status \"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6\": rpc error: code = NotFound desc = could not find container \"64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6\": container with ID starting with 64d2b178d38d0ed295e3c35e3a7a87f9c2c626e1cc46d9418383ab5d9755a3d6 not found: ID does not exist"
Jul 28 17:30:33.478505 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478457    2561 scope.go:117] "RemoveContainer" containerID="234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"
Jul 28 17:30:33.478658 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:33.478644    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6\": container with ID starting with 234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6 not found: ID does not exist" containerID="234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"
Jul 28 17:30:33.478701 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478660    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6"} err="failed to get container status \"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6\": rpc error: code = NotFound desc = could not find container \"234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6\": container with ID starting with 234f6f239e2198dbd6bb0bcea0b6f0814c6573db5f7c10e78fa09f24100bf8c6 not found: ID does not exist"
Jul 28 17:30:33.478701 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478672    2561 scope.go:117] "RemoveContainer" containerID="c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"
Jul 28 17:30:33.478909 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:33.478893    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42\": container with ID starting with c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42 not found: ID does not exist" containerID="c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"
Jul 28 17:30:33.478947 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42"} err="failed to get container status \"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42\": rpc error: code = NotFound desc = could not find container \"c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42\": container with ID starting with c2c357542db1ea0b378fedbd2489f248ee168ace241f453a04bfc7e979ce6f42 not found: ID does not exist"
Jul 28 17:30:33.478947 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.478927    2561 scope.go:117] "RemoveContainer" containerID="22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"
Jul 28 17:30:33.479165 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:30:33.479146    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1\": container with ID starting with 22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1 not found: ID does not exist" containerID="22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"
Jul 28 17:30:33.479209 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.479177    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1"} err="failed to get container status \"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1\": rpc error: code = NotFound desc = could not find container \"22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1\": container with ID starting with 22459ff29761fa5cdf0b29911a8accdfc7f85db6c232273a7667f91d96d53ec1 not found: ID does not exist"
Jul 28 17:30:33.537896 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.537841    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ce725fe-9f84-48cf-ac88-8429ce338c79-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:30:33.537896 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:33.537862    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpk9\" (UniqueName: \"kubernetes.io/projected/9ce725fe-9f84-48cf-ac88-8429ce338c79-kube-api-access-nzpk9\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:30:34.203317 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:34.203282    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" path="/var/lib/kubelet/pods/9ce725fe-9f84-48cf-ac88-8429ce338c79/volumes"
Jul 28 17:30:35.436939 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:35.436914    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:30:35.437626 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:35.437592    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:30:35.437788 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:35.437766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:45.438070 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:45.438025    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:30:45.438499 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:45.438477    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:30:55.438167 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:55.438123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:30:55.438616 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:30:55.438595    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:05.438400 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:05.438362    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:31:05.438919 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:05.438845    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:15.438044 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:15.438000    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:31:15.438488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:15.438418    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:25.437509 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:25.437463    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:31:25.437940 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:25.437894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:35.438048 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:35.438005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:31:35.438492 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:35.438469    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:45.438489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:45.438458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:31:45.438972 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:45.438551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:31:58.161793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.161766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-v67qd_0203403d-801b-4f2f-89a1-1b221827bae4/kserve-container/0.log"
Jul 28 17:31:58.416517 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.416432    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:31:58.416781 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.416729    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" containerID="cri-o://d3218608b39c9cf9596cd26a07bf15aece4c42e51281c23f802242e0216be9f6" gracePeriod=30
Jul 28 17:31:58.416781 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.416760    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" containerID="cri-o://518f892cf0e6320186f997d66550d8104a0a602a11b11746512d629f485925c8" gracePeriod=30
Jul 28 17:31:58.416970 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.416848    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" containerID="cri-o://b7678fcf982760ababdfb7bfb4c69d1c7d5d77c7943b2ebe88b5aba746af322b" gracePeriod=30
Jul 28 17:31:58.462087 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462058    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:31:58.462489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462491    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462507    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462515    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="storage-initializer"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462534    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="storage-initializer"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462547    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container"
Jul 28 17:31:58.462585 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462556    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container"
Jul 28 17:31:58.462927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462660    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="agent"
Jul 28 17:31:58.462927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462673    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kserve-container"
Jul 28 17:31:58.462927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.462686    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce725fe-9f84-48cf-ac88-8429ce338c79" containerName="kube-rbac-proxy"
Jul 28 17:31:58.466117 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.466097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.468910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.468892    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 17:31:58.468910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.468905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:31:58.475950 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.475928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:31:58.531533 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.531502    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:31:58.531737 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.531710    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kserve-container" containerID="cri-o://2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48" gracePeriod=30
Jul 28 17:31:58.531866 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.531770    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kube-rbac-proxy" containerID="cri-o://352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555" gracePeriod=30
Jul 28 17:31:58.545559 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.545532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48hk\" (UniqueName: \"kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.545666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.545561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.545666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.545586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.545666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.545641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.545787 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.545671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.646864 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.646838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.646983 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.646894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48hk\" (UniqueName: \"kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.646983 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.646910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.646983 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.646942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.647128 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.647055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.647378 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.647349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.647643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.647622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.647950 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.647917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.649200 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.649183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.655742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.655719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48hk\" (UniqueName: \"kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk\") pod \"isvc-autogluon-v1-predictor-574b5b7495-h2r48\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.774188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.774158    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:31:58.778351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.778318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:31:58.818821 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.818788    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerID="b7678fcf982760ababdfb7bfb4c69d1c7d5d77c7943b2ebe88b5aba746af322b" exitCode=2
Jul 28 17:31:58.819010 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.818858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerDied","Data":"b7678fcf982760ababdfb7bfb4c69d1c7d5d77c7943b2ebe88b5aba746af322b"}
Jul 28 17:31:58.821334 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821278    2561 generic.go:356] "Generic (PLEG): container finished" podID="0203403d-801b-4f2f-89a1-1b221827bae4" containerID="352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555" exitCode=2
Jul 28 17:31:58.821507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821340    2561 generic.go:356] "Generic (PLEG): container finished" podID="0203403d-801b-4f2f-89a1-1b221827bae4" containerID="2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48" exitCode=2
Jul 28 17:31:58.821507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerDied","Data":"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"}
Jul 28 17:31:58.821507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821475    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"
Jul 28 17:31:58.821507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821495    2561 scope.go:117] "RemoveContainer" containerID="352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"
Jul 28 17:31:58.821727 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerDied","Data":"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"}
Jul 28 17:31:58.821727 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.821619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd" event={"ID":"0203403d-801b-4f2f-89a1-1b221827bae4","Type":"ContainerDied","Data":"89d025baeaced2eb94e35f0874f80bc471e8201b90690b1c65cef19fa94f8cea"}
Jul 28 17:31:58.831192 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.830861    2561 scope.go:117] "RemoveContainer" containerID="2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"
Jul 28 17:31:58.842907 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.842873    2561 scope.go:117] "RemoveContainer" containerID="352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"
Jul 28 17:31:58.843419 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:31:58.843362    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555\": container with ID starting with 352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555 not found: ID does not exist" containerID="352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"
Jul 28 17:31:58.843419 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.843402    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"} err="failed to get container status \"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555\": rpc error: code = NotFound desc = could not find container \"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555\": container with ID starting with 352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555 not found: ID does not exist"
Jul 28 17:31:58.843611 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.843434    2561 scope.go:117] "RemoveContainer" containerID="2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"
Jul 28 17:31:58.843794 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:31:58.843768    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48\": container with ID starting with 2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48 not found: ID does not exist" containerID="2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"
Jul 28 17:31:58.843879 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.843806    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"} err="failed to get container status \"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48\": rpc error: code = NotFound desc = could not find container \"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48\": container with ID starting with 2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48 not found: ID does not exist"
Jul 28 17:31:58.843879 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.843829    2561 scope.go:117] "RemoveContainer" containerID="352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"
Jul 28 17:31:58.844148 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.844101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555"} err="failed to get container status \"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555\": rpc error: code = NotFound desc = could not find container \"352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555\": container with ID starting with 352dd77390fe262fd6f083c09529616236615dcd0bd746eb05bad34196f34555 not found: ID does not exist"
Jul 28 17:31:58.844148 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.844139    2561 scope.go:117] "RemoveContainer" containerID="2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"
Jul 28 17:31:58.844369 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.844346    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48"} err="failed to get container status \"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48\": rpc error: code = NotFound desc = could not find container \"2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48\": container with ID starting with 2f072e2a06bee3f286349b46e995e301a2478673e1aa37cc75fc6cb1c03e2f48 not found: ID does not exist"
Jul 28 17:31:58.848835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.848806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config\") pod \"0203403d-801b-4f2f-89a1-1b221827bae4\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") "
Jul 28 17:31:58.848976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.848879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqv8x\" (UniqueName: \"kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x\") pod \"0203403d-801b-4f2f-89a1-1b221827bae4\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") "
Jul 28 17:31:58.848976 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.848919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls\") pod \"0203403d-801b-4f2f-89a1-1b221827bae4\" (UID: \"0203403d-801b-4f2f-89a1-1b221827bae4\") "
Jul 28 17:31:58.849432 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.849293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0203403d-801b-4f2f-89a1-1b221827bae4" (UID: "0203403d-801b-4f2f-89a1-1b221827bae4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:31:58.851211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.851178    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0203403d-801b-4f2f-89a1-1b221827bae4" (UID: "0203403d-801b-4f2f-89a1-1b221827bae4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:58.851416 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.851235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x" (OuterVolumeSpecName: "kube-api-access-lqv8x") pod "0203403d-801b-4f2f-89a1-1b221827bae4" (UID: "0203403d-801b-4f2f-89a1-1b221827bae4"). InnerVolumeSpecName "kube-api-access-lqv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:58.918219 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.918098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:31:58.920932 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:31:58.920907    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3feb13_ce29_411d_82a6_6a2220041da3.slice/crio-238dcf05fe266c12290708711c6465a3226d077ec0265d698c3ecf5ac82e74db WatchSource:0}: Error finding container 238dcf05fe266c12290708711c6465a3226d077ec0265d698c3ecf5ac82e74db: Status 404 returned error can't find the container with id 238dcf05fe266c12290708711c6465a3226d077ec0265d698c3ecf5ac82e74db
Jul 28 17:31:58.950330 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.950290    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0203403d-801b-4f2f-89a1-1b221827bae4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:31:58.950330 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.950329    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqv8x\" (UniqueName: \"kubernetes.io/projected/0203403d-801b-4f2f-89a1-1b221827bae4-kube-api-access-lqv8x\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:31:58.950476 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:58.950340    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0203403d-801b-4f2f-89a1-1b221827bae4-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:31:59.143891 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:59.143862    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:31:59.147526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:59.147493    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd"]
Jul 28 17:31:59.828814 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:59.828729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerStarted","Data":"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"}
Jul 28 17:31:59.828814 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:31:59.828804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerStarted","Data":"238dcf05fe266c12290708711c6465a3226d077ec0265d698c3ecf5ac82e74db"}
Jul 28 17:32:00.205592 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:00.205560    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" path="/var/lib/kubelet/pods/0203403d-801b-4f2f-89a1-1b221827bae4/volumes"
Jul 28 17:32:00.433595 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:00.433549    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:03.852028 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:03.851938    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerID="d3218608b39c9cf9596cd26a07bf15aece4c42e51281c23f802242e0216be9f6" exitCode=0
Jul 28 17:32:03.852028 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:03.852016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerDied","Data":"d3218608b39c9cf9596cd26a07bf15aece4c42e51281c23f802242e0216be9f6"}
Jul 28 17:32:05.433081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:05.433034    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:05.437494 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:05.437467    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:32:05.437825 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:05.437800    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:10.432858 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:10.432807    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:10.433243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:10.432949    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:32:15.433286 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:15.433242    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:15.437636 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:15.437605    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:32:15.437990 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:15.437966    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:20.433434 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:20.433386    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:25.433138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:25.433088    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:32:25.437604 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:25.437569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:32:25.437723 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:25.437696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:32:25.437958 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:25.437935    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:25.438047 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:25.438035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:32:28.962155 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:28.962114    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerID="2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce" exitCode=0
Jul 28 17:32:28.963018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:28.962218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerDied","Data":"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"}
Jul 28 17:32:28.965167 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:28.965143    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerID="518f892cf0e6320186f997d66550d8104a0a602a11b11746512d629f485925c8" exitCode=0
Jul 28 17:32:28.965262 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:28.965204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerDied","Data":"518f892cf0e6320186f997d66550d8104a0a602a11b11746512d629f485925c8"}
Jul 28 17:32:29.073251 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.073223    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:32:29.113402 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert\") pod \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") "
Jul 28 17:32:29.113606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config\") pod \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") "
Jul 28 17:32:29.113606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location\") pod \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") "
Jul 28 17:32:29.113606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") pod \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") "
Jul 28 17:32:29.113606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113570    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwktr\" (UniqueName: \"kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr\") pod \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\" (UID: \"d2f33d36-a65b-42ba-a4c3-bdce22a26efa\") "
Jul 28 17:32:29.113891 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2f33d36-a65b-42ba-a4c3-bdce22a26efa" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:29.113891 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2f33d36-a65b-42ba-a4c3-bdce22a26efa" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:29.114080 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.113920    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2f33d36-a65b-42ba-a4c3-bdce22a26efa" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:29.115750 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.115727    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2f33d36-a65b-42ba-a4c3-bdce22a26efa" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:29.115845 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.115778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr" (OuterVolumeSpecName: "kube-api-access-rwktr") pod "d2f33d36-a65b-42ba-a4c3-bdce22a26efa" (UID: "d2f33d36-a65b-42ba-a4c3-bdce22a26efa"). InnerVolumeSpecName "kube-api-access-rwktr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:29.214366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.214251    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:32:29.214366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.214279    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:32:29.214366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.214289    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:32:29.214366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.214299    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:32:29.214366 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.214335    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwktr\" (UniqueName: \"kubernetes.io/projected/d2f33d36-a65b-42ba-a4c3-bdce22a26efa-kube-api-access-rwktr\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:32:29.973652 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.973620    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"
Jul 28 17:32:29.973652 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.973627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4" event={"ID":"d2f33d36-a65b-42ba-a4c3-bdce22a26efa","Type":"ContainerDied","Data":"7b722f23bcb01f02eba219845297eff1a78e2efe7826bafb39e15946609548be"}
Jul 28 17:32:29.974275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:29.973731    2561 scope.go:117] "RemoveContainer" containerID="518f892cf0e6320186f997d66550d8104a0a602a11b11746512d629f485925c8"
Jul 28 17:32:30.000418 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:30.000355    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:32:30.008823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:30.008785    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4"]
Jul 28 17:32:30.204288 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:30.204221    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" path="/var/lib/kubelet/pods/d2f33d36-a65b-42ba-a4c3-bdce22a26efa/volumes"
Jul 28 17:32:30.743411 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:30.743388    2561 scope.go:117] "RemoveContainer" containerID="b7678fcf982760ababdfb7bfb4c69d1c7d5d77c7943b2ebe88b5aba746af322b"
Jul 28 17:32:31.146940 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:31.146718    2561 scope.go:117] "RemoveContainer" containerID="d3218608b39c9cf9596cd26a07bf15aece4c42e51281c23f802242e0216be9f6"
Jul 28 17:32:31.251258 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:32:31.251237    2561 scope.go:117] "RemoveContainer" containerID="42379c282c51916d03f5c5d902958cbca87a7c2f0a38c4a9ea8d53f62c006412"
Jul 28 17:33:12.123444 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:33:12.123394    2561 scope.go:117] "RemoveContainer" containerID="bb25f662dc2cab375073535187f660889971aafabe62bd9f099b6484d3aefa89"
Jul 28 17:33:12.133081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:33:12.133042    2561 scope.go:117] "RemoveContainer" containerID="82011f82291d211835ed9e4564da6bfeb13defff619725f396ca6efc3b359c85"
Jul 28 17:33:12.141000 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:33:12.140978    2561 scope.go:117] "RemoveContainer" containerID="31283b399eca3ae638caf63efec5b0ebdb96c1160bdc188b0ceca1cd03b262d7"
Jul 28 17:33:12.149000 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:33:12.148977    2561 scope.go:117] "RemoveContainer" containerID="06d36d07a58e1f7021989816d17216282800c1da693468b432028d09beb1fcd8"
Jul 28 17:34:29.529142 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.529100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerStarted","Data":"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"}
Jul 28 17:34:29.529688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.529173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:29.529688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.529199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerStarted","Data":"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"}
Jul 28 17:34:29.529688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.529349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:29.530901 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.530871    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:34:29.557120 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:29.557075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podStartSLOduration=2.081851309 podStartE2EDuration="2m31.557060482s" totalImagesPullingTime="1m59.840755413s" totalInitContainerRuntime="29.63445376s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:29.555871798 +0000 UTC m=+1037.873310684" watchObservedRunningTime="2026-07-28 17:34:29.557060482 +0000 UTC m=+1037.874499437"
Jul 28 17:34:30.534094 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:30.534053    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:34:35.539744 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:35.539717    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:35.540214 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:35.540190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:34:45.541126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:45.541090    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:54.748556 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:54.748522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:34:54.749071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:54.748802    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container" containerID="cri-o://df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb" gracePeriod=30
Jul 28 17:34:54.749071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:54.748833    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kube-rbac-proxy" containerID="cri-o://50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431" gracePeriod=30
Jul 28 17:34:55.534974 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:55.534930    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 17:34:55.540198 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:55.540167    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:34:55.638090 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:55.638055    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerID="50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431" exitCode=2
Jul 28 17:34:55.638257 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:55.638092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerDied","Data":"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"}
Jul 28 17:34:56.124502 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.124478    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:56.295528 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295433    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls\") pod \"8a3feb13-ce29-411d-82a6-6a2220041da3\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") "
Jul 28 17:34:56.295528 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295529    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location\") pod \"8a3feb13-ce29-411d-82a6-6a2220041da3\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") "
Jul 28 17:34:56.295758 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert\") pod \"8a3feb13-ce29-411d-82a6-6a2220041da3\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") "
Jul 28 17:34:56.295758 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b48hk\" (UniqueName: \"kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk\") pod \"8a3feb13-ce29-411d-82a6-6a2220041da3\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") "
Jul 28 17:34:56.295758 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config\") pod \"8a3feb13-ce29-411d-82a6-6a2220041da3\" (UID: \"8a3feb13-ce29-411d-82a6-6a2220041da3\") "
Jul 28 17:34:56.295951 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.295927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a3feb13-ce29-411d-82a6-6a2220041da3" (UID: "8a3feb13-ce29-411d-82a6-6a2220041da3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:56.296041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.296018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a3feb13-ce29-411d-82a6-6a2220041da3" (UID: "8a3feb13-ce29-411d-82a6-6a2220041da3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:56.297599 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.297571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a3feb13-ce29-411d-82a6-6a2220041da3" (UID: "8a3feb13-ce29-411d-82a6-6a2220041da3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:56.297971 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.297952    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk" (OuterVolumeSpecName: "kube-api-access-b48hk") pod "8a3feb13-ce29-411d-82a6-6a2220041da3" (UID: "8a3feb13-ce29-411d-82a6-6a2220041da3"). InnerVolumeSpecName "kube-api-access-b48hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:56.300827 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.300804    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a3feb13-ce29-411d-82a6-6a2220041da3" (UID: "8a3feb13-ce29-411d-82a6-6a2220041da3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:56.397140 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.397113    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3feb13-ce29-411d-82a6-6a2220041da3-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.397140 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.397136    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.397361 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.397149    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48hk\" (UniqueName: \"kubernetes.io/projected/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-api-access-b48hk\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.397361 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.397161    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3feb13-ce29-411d-82a6-6a2220041da3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.397361 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.397170    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3feb13-ce29-411d-82a6-6a2220041da3-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.644428 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.644345    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerID="df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb" exitCode=0
Jul 28 17:34:56.644428 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.644398    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"
Jul 28 17:34:56.644631 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.644435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerDied","Data":"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"}
Jul 28 17:34:56.644631 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.644491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48" event={"ID":"8a3feb13-ce29-411d-82a6-6a2220041da3","Type":"ContainerDied","Data":"238dcf05fe266c12290708711c6465a3226d077ec0265d698c3ecf5ac82e74db"}
Jul 28 17:34:56.644631 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.644519    2561 scope.go:117] "RemoveContainer" containerID="50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"
Jul 28 17:34:56.653046 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.653029    2561 scope.go:117] "RemoveContainer" containerID="df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"
Jul 28 17:34:56.660627 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.660611    2561 scope.go:117] "RemoveContainer" containerID="2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"
Jul 28 17:34:56.666264 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.666243    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:34:56.667704 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.667687    2561 scope.go:117] "RemoveContainer" containerID="50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"
Jul 28 17:34:56.668009 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:34:56.667985    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431\": container with ID starting with 50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431 not found: ID does not exist" containerID="50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"
Jul 28 17:34:56.668092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.668049    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431"} err="failed to get container status \"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431\": rpc error: code = NotFound desc = could not find container \"50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431\": container with ID starting with 50fd8062b6805388990a8ad04ce9152345ff353915d3d8425ea3bc7d5c9d4431 not found: ID does not exist"
Jul 28 17:34:56.668092 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.668076    2561 scope.go:117] "RemoveContainer" containerID="df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"
Jul 28 17:34:56.668512 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:34:56.668491    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb\": container with ID starting with df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb not found: ID does not exist" containerID="df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"
Jul 28 17:34:56.668609 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.668518    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb"} err="failed to get container status \"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb\": rpc error: code = NotFound desc = could not find container \"df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb\": container with ID starting with df3e78ae8e88cc2b17cb57479b50dfab0cbd7644c6d60458c99da32d5f5b76eb not found: ID does not exist"
Jul 28 17:34:56.668609 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.668536    2561 scope.go:117] "RemoveContainer" containerID="2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"
Jul 28 17:34:56.668808 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:34:56.668791    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce\": container with ID starting with 2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce not found: ID does not exist" containerID="2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"
Jul 28 17:34:56.668866 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.668813    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce"} err="failed to get container status \"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce\": rpc error: code = NotFound desc = could not find container \"2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce\": container with ID starting with 2f95a1efd38064460aa326d5d0d86095a07ab9ef3cf2b2dac88e9821257b8fce not found: ID does not exist"
Jul 28 17:34:56.670020 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:56.669999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48"]
Jul 28 17:34:58.203201 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:58.203172    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" path="/var/lib/kubelet/pods/8a3feb13-ce29-411d-82a6-6a2220041da3/volumes"
Jul 28 17:34:59.934635 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.934596    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935035    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="storage-initializer"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935053    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="storage-initializer"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935066    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935073    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935083    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935091    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935109    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935123    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kserve-container"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935129    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kserve-container"
Jul 28 17:34:59.935138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935140    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935149    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935158    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="storage-initializer"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935166    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="storage-initializer"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935183    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935191    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935217    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935286    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935326    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="agent"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935334    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kserve-container"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935344    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0203403d-801b-4f2f-89a1-1b221827bae4" containerName="kube-rbac-proxy"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935351    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f33d36-a65b-42ba-a4c3-bdce22a26efa" containerName="kserve-container"
Jul 28 17:34:59.935725 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.935359    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3feb13-ce29-411d-82a6-6a2220041da3" containerName="kserve-container"
Jul 28 17:34:59.954237 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.954200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:34:59.954419 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.954340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:34:59.956917 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.956895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 17:34:59.957039 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.956943    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:34:59.957174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.957161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 17:34:59.957241 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.957225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:34:59.957880 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.957865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:34:59.958065 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.958052    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:34:59.958175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:34:59.958158    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:35:00.025982 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.025959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.026108 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.025995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.026108 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.026018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.026108 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.026067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp44\" (UniqueName: \"kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.026108 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.026085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126489 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp44\" (UniqueName: \"kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.126827 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:35:00.126808    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 17:35:00.126900 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:35:00.126887    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls podName:ade4c2c2-efff-4524-8a73-c642d3466978 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:00.626864093 +0000 UTC m=+1068.944302963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 17:35:00.126900 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.126879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.127176 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.127153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.127176 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.127171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.135540 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.135516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp44\" (UniqueName: \"kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.630113 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.630082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.632369 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.632351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wt6jh\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:00.865426 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:00.865391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:01.023957 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:01.023933    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:35:01.026090 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:35:01.026068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade4c2c2_efff_4524_8a73_c642d3466978.slice/crio-d54597120c27c2d39e9f90265a9bfafdfe2a7a8d2a5c59f0fc9231756518c283 WatchSource:0}: Error finding container d54597120c27c2d39e9f90265a9bfafdfe2a7a8d2a5c59f0fc9231756518c283: Status 404 returned error can't find the container with id d54597120c27c2d39e9f90265a9bfafdfe2a7a8d2a5c59f0fc9231756518c283
Jul 28 17:35:01.666663 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:01.666621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerStarted","Data":"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"}
Jul 28 17:35:01.666843 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:01.666693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerStarted","Data":"d54597120c27c2d39e9f90265a9bfafdfe2a7a8d2a5c59f0fc9231756518c283"}
Jul 28 17:35:06.688200 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:06.688162    2561 generic.go:356] "Generic (PLEG): container finished" podID="ade4c2c2-efff-4524-8a73-c642d3466978" containerID="8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f" exitCode=0
Jul 28 17:35:06.688713 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:06.688210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerDied","Data":"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"}
Jul 28 17:35:07.693816 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:07.693780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerStarted","Data":"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"}
Jul 28 17:35:07.694325 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:07.693844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerStarted","Data":"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"}
Jul 28 17:35:07.694325 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:07.694027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:07.714010 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:07.713959    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" podStartSLOduration=3.686697787 podStartE2EDuration="8.713945281s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.027247494s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:07.712146455 +0000 UTC m=+1076.029585342" watchObservedRunningTime="2026-07-28 17:35:07.713945281 +0000 UTC m=+1076.031384166"
Jul 28 17:35:08.698235 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:08.698189    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:35:14.708051 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:35:14.708017    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:36:43.201951 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:43.201861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:36:54.239515 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:54.239474    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:36:56.125271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:56.125236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:36:56.125740 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:56.125476    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kserve-container" containerID="cri-o://257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d" gracePeriod=30
Jul 28 17:36:56.125740 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:56.125522    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kube-rbac-proxy" containerID="cri-o://54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a" gracePeriod=30
Jul 28 17:36:57.126409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.126376    2561 generic.go:356] "Generic (PLEG): container finished" podID="ade4c2c2-efff-4524-8a73-c642d3466978" containerID="54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a" exitCode=2
Jul 28 17:36:57.126801 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.126451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerDied","Data":"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"}
Jul 28 17:36:57.563987 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.563959    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:36:57.638114 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") pod \"ade4c2c2-efff-4524-8a73-c642d3466978\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") "
Jul 28 17:36:57.638293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config\") pod \"ade4c2c2-efff-4524-8a73-c642d3466978\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") "
Jul 28 17:36:57.638293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location\") pod \"ade4c2c2-efff-4524-8a73-c642d3466978\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") "
Jul 28 17:36:57.638293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638232    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert\") pod \"ade4c2c2-efff-4524-8a73-c642d3466978\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") "
Jul 28 17:36:57.638293 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fp44\" (UniqueName: \"kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44\") pod \"ade4c2c2-efff-4524-8a73-c642d3466978\" (UID: \"ade4c2c2-efff-4524-8a73-c642d3466978\") "
Jul 28 17:36:57.638537 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ade4c2c2-efff-4524-8a73-c642d3466978" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:57.638684 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.638657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ade4c2c2-efff-4524-8a73-c642d3466978" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:57.640261 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.640235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44" (OuterVolumeSpecName: "kube-api-access-4fp44") pod "ade4c2c2-efff-4524-8a73-c642d3466978" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978"). InnerVolumeSpecName "kube-api-access-4fp44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:57.640261 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.640258    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ade4c2c2-efff-4524-8a73-c642d3466978" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:57.642880 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.642859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ade4c2c2-efff-4524-8a73-c642d3466978" (UID: "ade4c2c2-efff-4524-8a73-c642d3466978"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:57.739927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.739847    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade4c2c2-efff-4524-8a73-c642d3466978-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:36:57.739927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.739878    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:36:57.739927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.739891    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fp44\" (UniqueName: \"kubernetes.io/projected/ade4c2c2-efff-4524-8a73-c642d3466978-kube-api-access-4fp44\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:36:57.739927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.739901    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade4c2c2-efff-4524-8a73-c642d3466978-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:36:57.739927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:57.739911    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade4c2c2-efff-4524-8a73-c642d3466978-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:36:58.132688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.132601    2561 generic.go:356] "Generic (PLEG): container finished" podID="ade4c2c2-efff-4524-8a73-c642d3466978" containerID="257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d" exitCode=0
Jul 28 17:36:58.132688 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.132662    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"
Jul 28 17:36:58.133188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.132664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerDied","Data":"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"}
Jul 28 17:36:58.133188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.132810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh" event={"ID":"ade4c2c2-efff-4524-8a73-c642d3466978","Type":"ContainerDied","Data":"d54597120c27c2d39e9f90265a9bfafdfe2a7a8d2a5c59f0fc9231756518c283"}
Jul 28 17:36:58.133188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.132838    2561 scope.go:117] "RemoveContainer" containerID="54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"
Jul 28 17:36:58.143410 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.143379    2561 scope.go:117] "RemoveContainer" containerID="257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"
Jul 28 17:36:58.150844 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.150826    2561 scope.go:117] "RemoveContainer" containerID="8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"
Jul 28 17:36:58.157222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.157200    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:36:58.158122 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.158107    2561 scope.go:117] "RemoveContainer" containerID="54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"
Jul 28 17:36:58.158418 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:36:58.158398    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a\": container with ID starting with 54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a not found: ID does not exist" containerID="54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"
Jul 28 17:36:58.158507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.158431    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a"} err="failed to get container status \"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a\": rpc error: code = NotFound desc = could not find container \"54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a\": container with ID starting with 54fcfe66fbce6c32c256ac381f8f90ac0563d9c4ffd41c11b99ae759a471381a not found: ID does not exist"
Jul 28 17:36:58.158507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.158455    2561 scope.go:117] "RemoveContainer" containerID="257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"
Jul 28 17:36:58.158725 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:36:58.158706    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d\": container with ID starting with 257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d not found: ID does not exist" containerID="257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"
Jul 28 17:36:58.158765 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.158731    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d"} err="failed to get container status \"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d\": rpc error: code = NotFound desc = could not find container \"257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d\": container with ID starting with 257b6be3fa55a9b988679cd838d82a67c72a0849dfbc5c921751deb73c308b8d not found: ID does not exist"
Jul 28 17:36:58.158765 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.158752    2561 scope.go:117] "RemoveContainer" containerID="8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"
Jul 28 17:36:58.159009 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:36:58.158984    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f\": container with ID starting with 8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f not found: ID does not exist" containerID="8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"
Jul 28 17:36:58.159061 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.159008    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f"} err="failed to get container status \"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f\": rpc error: code = NotFound desc = could not find container \"8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f\": container with ID starting with 8aa91341045494e6200f77e4c3eac206ec19936f6cf4a0fd85660aa94124537f not found: ID does not exist"
Jul 28 17:36:58.161760 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.161740    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh"]
Jul 28 17:36:58.203553 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:36:58.203532    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" path="/var/lib/kubelet/pods/ade4c2c2-efff-4524-8a73-c642d3466978/volumes"
Jul 28 17:37:01.274565 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274538    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274847    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kube-rbac-proxy"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274857    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kube-rbac-proxy"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274866    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="storage-initializer"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274872    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="storage-initializer"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274883    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kserve-container"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274888    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kserve-container"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274954    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kserve-container"
Jul 28 17:37:01.275008 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.274960    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade4c2c2-efff-4524-8a73-c642d3466978" containerName="kube-rbac-proxy"
Jul 28 17:37:01.278436 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.278412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.280856 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.280834    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:37:01.280982 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.280924    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 17:37:01.280982 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.280947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 17:37:01.280982 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.280958    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:37:01.280982 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.280953    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:37:01.281779 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.281765    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:37:01.281779 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.281773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:37:01.289199 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.289179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:37:01.368930 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.368890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.368930 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.368929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.369112 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.368955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.369112 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.369029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw8c\" (UniqueName: \"kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.369187 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.369114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.470504 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.470471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.470657 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.470527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.470657 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.470560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.470657 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.470584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.470657 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.470628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw8c\" (UniqueName: \"kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.471054 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.471021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.471190 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.471172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.471251 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.471232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.473155 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.473134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.478242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.478212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw8c\" (UniqueName: \"kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.588669 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.588587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:01.716572 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:01.716539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:37:01.719268 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:37:01.719233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1a3751_c666_40b4_a3c2_c70df478a32f.slice/crio-e2cb9ffc2e8a9fb8e375edcf5d83c54fbf76a90c4fc1c3646c5e017c200665a1 WatchSource:0}: Error finding container e2cb9ffc2e8a9fb8e375edcf5d83c54fbf76a90c4fc1c3646c5e017c200665a1: Status 404 returned error can't find the container with id e2cb9ffc2e8a9fb8e375edcf5d83c54fbf76a90c4fc1c3646c5e017c200665a1
Jul 28 17:37:02.150564 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:02.150527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerStarted","Data":"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"}
Jul 28 17:37:02.150745 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:02.150616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerStarted","Data":"e2cb9ffc2e8a9fb8e375edcf5d83c54fbf76a90c4fc1c3646c5e017c200665a1"}
Jul 28 17:37:07.173061 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:07.173030    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerID="9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f" exitCode=0
Jul 28 17:37:07.173442 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:07.173074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerDied","Data":"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"}
Jul 28 17:37:08.179208 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:08.179171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerStarted","Data":"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"}
Jul 28 17:37:08.179584 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:08.179270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerStarted","Data":"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"}
Jul 28 17:37:08.179584 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:08.179489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:08.200892 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:08.200839    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" podStartSLOduration=2.182292072 podStartE2EDuration="7.200818328s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.018526256s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:08.200151755 +0000 UTC m=+1196.517590644" watchObservedRunningTime="2026-07-28 17:37:08.200818328 +0000 UTC m=+1196.518257217"
Jul 28 17:37:09.187363 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:09.187331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:37:15.196431 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:37:15.196404    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:38:42.204680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:42.204647    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:38:53.242465 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:53.242420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:38:57.880842 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:57.880810    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:38:57.881290 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:57.881090    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kserve-container" containerID="cri-o://08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9" gracePeriod=30
Jul 28 17:38:57.881290 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:57.881150    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kube-rbac-proxy" containerID="cri-o://8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6" gracePeriod=30
Jul 28 17:38:58.620401 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:58.620368    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerID="8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6" exitCode=2
Jul 28 17:38:58.620660 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:58.620439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerDied","Data":"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"}
Jul 28 17:38:59.230790 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.230766    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:38:59.335111 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config\") pod \"ae1a3751-c666-40b4-a3c2-c70df478a32f\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") "
Jul 28 17:38:59.335111 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert\") pod \"ae1a3751-c666-40b4-a3c2-c70df478a32f\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") "
Jul 28 17:38:59.335111 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls\") pod \"ae1a3751-c666-40b4-a3c2-c70df478a32f\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") "
Jul 28 17:38:59.335411 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location\") pod \"ae1a3751-c666-40b4-a3c2-c70df478a32f\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") "
Jul 28 17:38:59.335411 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335224    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfw8c\" (UniqueName: \"kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c\") pod \"ae1a3751-c666-40b4-a3c2-c70df478a32f\" (UID: \"ae1a3751-c666-40b4-a3c2-c70df478a32f\") "
Jul 28 17:38:59.335517 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335428    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae1a3751-c666-40b4-a3c2-c70df478a32f" (UID: "ae1a3751-c666-40b4-a3c2-c70df478a32f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:59.335517 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae1a3751-c666-40b4-a3c2-c70df478a32f" (UID: "ae1a3751-c666-40b4-a3c2-c70df478a32f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:59.335629 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.335517    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:38:59.337143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.337115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae1a3751-c666-40b4-a3c2-c70df478a32f" (UID: "ae1a3751-c666-40b4-a3c2-c70df478a32f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:59.337317 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.337282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c" (OuterVolumeSpecName: "kube-api-access-cfw8c") pod "ae1a3751-c666-40b4-a3c2-c70df478a32f" (UID: "ae1a3751-c666-40b4-a3c2-c70df478a32f"). InnerVolumeSpecName "kube-api-access-cfw8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:59.341620 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.341591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae1a3751-c666-40b4-a3c2-c70df478a32f" (UID: "ae1a3751-c666-40b4-a3c2-c70df478a32f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:59.436678 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.436646    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae1a3751-c666-40b4-a3c2-c70df478a32f-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:38:59.436678 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.436672    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae1a3751-c666-40b4-a3c2-c70df478a32f-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:38:59.436678 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.436682    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae1a3751-c666-40b4-a3c2-c70df478a32f-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:38:59.436895 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.436691    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw8c\" (UniqueName: \"kubernetes.io/projected/ae1a3751-c666-40b4-a3c2-c70df478a32f-kube-api-access-cfw8c\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:38:59.626064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.625985    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerID="08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9" exitCode=0
Jul 28 17:38:59.626064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.626026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerDied","Data":"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"}
Jul 28 17:38:59.626064 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.626052    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"
Jul 28 17:38:59.626326 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.626073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8" event={"ID":"ae1a3751-c666-40b4-a3c2-c70df478a32f","Type":"ContainerDied","Data":"e2cb9ffc2e8a9fb8e375edcf5d83c54fbf76a90c4fc1c3646c5e017c200665a1"}
Jul 28 17:38:59.626326 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.626099    2561 scope.go:117] "RemoveContainer" containerID="8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"
Jul 28 17:38:59.634464 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.634450    2561 scope.go:117] "RemoveContainer" containerID="08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"
Jul 28 17:38:59.641595 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.641576    2561 scope.go:117] "RemoveContainer" containerID="9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"
Jul 28 17:38:59.647353 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.647328    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:38:59.648749 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.648731    2561 scope.go:117] "RemoveContainer" containerID="8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"
Jul 28 17:38:59.648981 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:38:59.648963    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6\": container with ID starting with 8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6 not found: ID does not exist" containerID="8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"
Jul 28 17:38:59.649027 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.648992    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6"} err="failed to get container status \"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6\": rpc error: code = NotFound desc = could not find container \"8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6\": container with ID starting with 8298044ae714a7d7eb6fc07775900921cb069604be25d68330dfc2e8d1b0b0e6 not found: ID does not exist"
Jul 28 17:38:59.649027 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.649009    2561 scope.go:117] "RemoveContainer" containerID="08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"
Jul 28 17:38:59.649230 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:38:59.649216    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9\": container with ID starting with 08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9 not found: ID does not exist" containerID="08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"
Jul 28 17:38:59.649270 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.649233    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9"} err="failed to get container status \"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9\": rpc error: code = NotFound desc = could not find container \"08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9\": container with ID starting with 08da332a70a6b271a34805d56a78f1a89384fc1e6a8754700c173efd712a5dd9 not found: ID does not exist"
Jul 28 17:38:59.649270 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.649247    2561 scope.go:117] "RemoveContainer" containerID="9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"
Jul 28 17:38:59.649526 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:38:59.649506    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f\": container with ID starting with 9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f not found: ID does not exist" containerID="9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"
Jul 28 17:38:59.649578 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.649530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f"} err="failed to get container status \"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f\": rpc error: code = NotFound desc = could not find container \"9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f\": container with ID starting with 9c2e67589d3d3fc9f9d4c2434602d37f8cffdfd0fff367f2443a638ef31baa9f not found: ID does not exist"
Jul 28 17:38:59.653043 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:38:59.653022    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8"]
Jul 28 17:39:00.203886 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:00.203852    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" path="/var/lib/kubelet/pods/ae1a3751-c666-40b4-a3c2-c70df478a32f/volumes"
Jul 28 17:39:03.020138 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020105    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020435    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="storage-initializer"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020446    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="storage-initializer"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020456    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kube-rbac-proxy"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020461    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kube-rbac-proxy"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020474    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kserve-container"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020480    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kserve-container"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020529    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kube-rbac-proxy"
Jul 28 17:39:03.020563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.020539    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1a3751-c666-40b4-a3c2-c70df478a32f" containerName="kserve-container"
Jul 28 17:39:03.025431 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.025407    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.027754 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.027731    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:39:03.028041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:39:03.028137 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:39:03.028137 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028096    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:39:03.028226 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 17:39:03.028762 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:39:03.028865 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.028848    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:39:03.033969 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.033952    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:39:03.166085 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.166052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.166254 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.166092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.166254 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.166112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbs5\" (UniqueName: \"kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.166254 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.166197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.166404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.166254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267431 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbs5\" (UniqueName: \"kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.267602 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:39:03.267549    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 28 17:39:03.267867 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:39:03.267621    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls podName:056ce988-1604-489f-b308-0fb6d2ec92c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:39:03.767601174 +0000 UTC m=+1312.085040060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 28 17:39:03.267938 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.267895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.268140 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.268121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.268239 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.268218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.277180 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.277124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbs5\" (UniqueName: \"kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.770942 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.770906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.773224 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.773194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:03.936814 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:03.936753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:04.065411 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:04.065389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:39:04.067801 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:39:04.067774    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056ce988_1604_489f_b308_0fb6d2ec92c1.slice/crio-cce362baa81fc13f5fe051abba3c09a9aee339d44a61172813171c13b1c2d56f WatchSource:0}: Error finding container cce362baa81fc13f5fe051abba3c09a9aee339d44a61172813171c13b1c2d56f: Status 404 returned error can't find the container with id cce362baa81fc13f5fe051abba3c09a9aee339d44a61172813171c13b1c2d56f
Jul 28 17:39:04.649642 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:04.649606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerStarted","Data":"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"}
Jul 28 17:39:04.649842 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:04.649674    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerStarted","Data":"cce362baa81fc13f5fe051abba3c09a9aee339d44a61172813171c13b1c2d56f"}
Jul 28 17:39:09.671705 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:09.671667    2561 generic.go:356] "Generic (PLEG): container finished" podID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerID="7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562" exitCode=0
Jul 28 17:39:09.672169 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:09.671740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerDied","Data":"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"}
Jul 28 17:39:10.677816 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:10.677783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerStarted","Data":"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"}
Jul 28 17:39:10.678189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:10.677844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerStarted","Data":"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"}
Jul 28 17:39:10.678189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:10.678021    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:10.699617 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:10.699569    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" podStartSLOduration=2.967922777 podStartE2EDuration="7.699553766s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.731630989s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:10.697925032 +0000 UTC m=+1319.015363919" watchObservedRunningTime="2026-07-28 17:39:10.699553766 +0000 UTC m=+1319.016992652"
Jul 28 17:39:11.682807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:11.682775    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:39:17.691515 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:39:17.691489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:40:39.202883 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:39.202847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:40:49.262394 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:49.262346    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:40:49.262848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:49.262656    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kserve-container" containerID="cri-o://7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635" gracePeriod=30
Jul 28 17:40:49.262848 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:49.262776    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kube-rbac-proxy" containerID="cri-o://9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f" gracePeriod=30
Jul 28 17:40:50.069891 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.069854    2561 generic.go:356] "Generic (PLEG): container finished" podID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerID="9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f" exitCode=2
Jul 28 17:40:50.070066 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.069917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerDied","Data":"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"}
Jul 28 17:40:50.612271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.612251    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:40:50.742519 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location\") pod \"056ce988-1604-489f-b308-0fb6d2ec92c1\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") "
Jul 28 17:40:50.742519 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") pod \"056ce988-1604-489f-b308-0fb6d2ec92c1\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") "
Jul 28 17:40:50.742757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742530    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config\") pod \"056ce988-1604-489f-b308-0fb6d2ec92c1\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") "
Jul 28 17:40:50.742757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwbs5\" (UniqueName: \"kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5\") pod \"056ce988-1604-489f-b308-0fb6d2ec92c1\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") "
Jul 28 17:40:50.742757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert\") pod \"056ce988-1604-489f-b308-0fb6d2ec92c1\" (UID: \"056ce988-1604-489f-b308-0fb6d2ec92c1\") "
Jul 28 17:40:50.742959 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.742930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "056ce988-1604-489f-b308-0fb6d2ec92c1" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:50.743073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.743046    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "056ce988-1604-489f-b308-0fb6d2ec92c1" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:50.744619 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.744596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "056ce988-1604-489f-b308-0fb6d2ec92c1" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:50.744723 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.744690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5" (OuterVolumeSpecName: "kube-api-access-wwbs5") pod "056ce988-1604-489f-b308-0fb6d2ec92c1" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1"). InnerVolumeSpecName "kube-api-access-wwbs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:50.747373 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.747349    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "056ce988-1604-489f-b308-0fb6d2ec92c1" (UID: "056ce988-1604-489f-b308-0fb6d2ec92c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:50.843863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.843814    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:40:50.843863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.843858    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056ce988-1604-489f-b308-0fb6d2ec92c1-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:40:50.843863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.843869    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/056ce988-1604-489f-b308-0fb6d2ec92c1-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:40:50.843863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.843879    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:40:50.844105 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:50.843887    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbs5\" (UniqueName: \"kubernetes.io/projected/056ce988-1604-489f-b308-0fb6d2ec92c1-kube-api-access-wwbs5\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:40:51.075243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.075154    2561 generic.go:356] "Generic (PLEG): container finished" podID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerID="7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635" exitCode=0
Jul 28 17:40:51.075243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.075212    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"
Jul 28 17:40:51.075243 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.075233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerDied","Data":"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"}
Jul 28 17:40:51.075484 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.075293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx" event={"ID":"056ce988-1604-489f-b308-0fb6d2ec92c1","Type":"ContainerDied","Data":"cce362baa81fc13f5fe051abba3c09a9aee339d44a61172813171c13b1c2d56f"}
Jul 28 17:40:51.075484 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.075337    2561 scope.go:117] "RemoveContainer" containerID="9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"
Jul 28 17:40:51.088011 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.087984    2561 scope.go:117] "RemoveContainer" containerID="7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"
Jul 28 17:40:51.095752 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.095733    2561 scope.go:117] "RemoveContainer" containerID="7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"
Jul 28 17:40:51.101405 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.101380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:40:51.103189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.103167    2561 scope.go:117] "RemoveContainer" containerID="9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"
Jul 28 17:40:51.103569 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:40:51.103545    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f\": container with ID starting with 9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f not found: ID does not exist" containerID="9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"
Jul 28 17:40:51.103752 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.103727    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f"} err="failed to get container status \"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f\": rpc error: code = NotFound desc = could not find container \"9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f\": container with ID starting with 9032be403b241ca916609145ce75c6cd19ba3c8ecbe2df3d4f7fbda1a57e705f not found: ID does not exist"
Jul 28 17:40:51.103861 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.103843    2561 scope.go:117] "RemoveContainer" containerID="7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"
Jul 28 17:40:51.104156 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:40:51.104139    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635\": container with ID starting with 7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635 not found: ID does not exist" containerID="7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"
Jul 28 17:40:51.104226 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.104162    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635"} err="failed to get container status \"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635\": rpc error: code = NotFound desc = could not find container \"7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635\": container with ID starting with 7e40a0bbfd67cb1579f4b6b58114ef394374ab9c338e6a6a5e6244a9941d7635 not found: ID does not exist"
Jul 28 17:40:51.104226 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.104177    2561 scope.go:117] "RemoveContainer" containerID="7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"
Jul 28 17:40:51.104901 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:40:51.104881    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562\": container with ID starting with 7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562 not found: ID does not exist" containerID="7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"
Jul 28 17:40:51.104968 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.104905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562"} err="failed to get container status \"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562\": rpc error: code = NotFound desc = could not find container \"7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562\": container with ID starting with 7fc7db924b8cc90b44b0d235a15b996c52f6965abdf5844311cd83b61c2f0562 not found: ID does not exist"
Jul 28 17:40:51.105286 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:51.105268    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx"]
Jul 28 17:40:52.202813 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:52.202781    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" path="/var/lib/kubelet/pods/056ce988-1604-489f-b308-0fb6d2ec92c1/volumes"
Jul 28 17:40:54.459668 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.459631    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460057    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kube-rbac-proxy"
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460073    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kube-rbac-proxy"
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="storage-initializer"
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460101    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="storage-initializer"
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460110    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kserve-container"
Jul 28 17:40:54.460115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460118    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kserve-container"
Jul 28 17:40:54.460465 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460194    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kube-rbac-proxy"
Jul 28 17:40:54.460465 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.460211    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="056ce988-1604-489f-b308-0fb6d2ec92c1" containerName="kserve-container"
Jul 28 17:40:54.465074 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.465053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.467892 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.467873    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:40:54.468295 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:40:54.468391 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468317    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:40:54.468391 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468333    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:40:54.468958 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468943    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:40:54.469052 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468956    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:54.469052 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.468947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 17:40:54.473562 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.473543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:40:54.572351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.572296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.572351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.572354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.572557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.572425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8hk\" (UniqueName: \"kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.572557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.572495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.572557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.572539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673245 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8hk\" (UniqueName: \"kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.673558 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:40:54.673457    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 17:40:54.673558 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:40:54.673537    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls podName:ffcac168-22d4-4a17-8dba-a0c7388e69a0 nodeName:}" failed. No retries permitted until 2026-07-28 17:40:55.173520554 +0000 UTC m=+1423.490959419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 17:40:54.673753 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.673731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.674037 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.674011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.674115 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.674101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:54.684444 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:54.684417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8hk\" (UniqueName: \"kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:55.177715 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:55.177681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:55.180024 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:55.179992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:55.375556 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:55.375514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:40:55.503928 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:55.503907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:40:55.506258 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:40:55.506233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcac168_22d4_4a17_8dba_a0c7388e69a0.slice/crio-e9c1399c07f22892c28bf7c65a3f5c2c19454327f5ea945d49afd6dbe9a71d2f WatchSource:0}: Error finding container e9c1399c07f22892c28bf7c65a3f5c2c19454327f5ea945d49afd6dbe9a71d2f: Status 404 returned error can't find the container with id e9c1399c07f22892c28bf7c65a3f5c2c19454327f5ea945d49afd6dbe9a71d2f
Jul 28 17:40:56.097702 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:56.097667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerStarted","Data":"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"}
Jul 28 17:40:56.097912 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:40:56.097746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerStarted","Data":"e9c1399c07f22892c28bf7c65a3f5c2c19454327f5ea945d49afd6dbe9a71d2f"}
Jul 28 17:41:04.131012 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:04.130924    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerID="0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28" exitCode=0
Jul 28 17:41:04.131012 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:04.130966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerDied","Data":"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"}
Jul 28 17:41:05.138432 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.138400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerStarted","Data":"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"}
Jul 28 17:41:05.138881 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.138458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerStarted","Data":"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"}
Jul 28 17:41:05.138881 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.138731    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:05.139012 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.138902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:05.139927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.139894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:41:05.161437 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:05.161374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podStartSLOduration=2.934516574 podStartE2EDuration="11.161355282s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.226838708s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:05.15983517 +0000 UTC m=+1433.477274084" watchObservedRunningTime="2026-07-28 17:41:05.161355282 +0000 UTC m=+1433.478794167"
Jul 28 17:41:06.143262 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:06.143214    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:41:11.147744 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:11.147712    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:11.148382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:11.148354    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:41:21.149463 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:21.149437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:36.938986 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:36.938950    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:41:36.939430 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:36.939251    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container" containerID="cri-o://14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396" gracePeriod=30
Jul 28 17:41:36.939430 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:36.939285    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kube-rbac-proxy" containerID="cri-o://63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879" gracePeriod=30
Jul 28 17:41:37.268948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:37.268862    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerID="63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879" exitCode=2
Jul 28 17:41:37.268948 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:37.268927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerDied","Data":"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"}
Jul 28 17:41:38.689144 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.689118    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:38.870645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870553    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert\") pod \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") "
Jul 28 17:41:38.870645 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config\") pod \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") "
Jul 28 17:41:38.870875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870685    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w8hk\" (UniqueName: \"kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk\") pod \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") "
Jul 28 17:41:38.870875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870716    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location\") pod \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") "
Jul 28 17:41:38.870875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") pod \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\" (UID: \"ffcac168-22d4-4a17-8dba-a0c7388e69a0\") "
Jul 28 17:41:38.871065 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.870980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffcac168-22d4-4a17-8dba-a0c7388e69a0" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:38.871065 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.871045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffcac168-22d4-4a17-8dba-a0c7388e69a0" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:38.871767 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.871746    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffcac168-22d4-4a17-8dba-a0c7388e69a0" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:38.872772 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.872749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk" (OuterVolumeSpecName: "kube-api-access-5w8hk") pod "ffcac168-22d4-4a17-8dba-a0c7388e69a0" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0"). InnerVolumeSpecName "kube-api-access-5w8hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:38.872857 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.872837    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffcac168-22d4-4a17-8dba-a0c7388e69a0" (UID: "ffcac168-22d4-4a17-8dba-a0c7388e69a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:38.971666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.971633    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:41:38.971666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.971661    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:41:38.971666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.971670    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8hk\" (UniqueName: \"kubernetes.io/projected/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kube-api-access-5w8hk\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:41:38.971903 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.971680    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffcac168-22d4-4a17-8dba-a0c7388e69a0-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:41:38.971903 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:38.971690    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffcac168-22d4-4a17-8dba-a0c7388e69a0-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:41:39.278113 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.278079    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerID="14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396" exitCode=0
Jul 28 17:41:39.278341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.278137    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"
Jul 28 17:41:39.278341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.278171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerDied","Data":"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"}
Jul 28 17:41:39.278341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.278245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5" event={"ID":"ffcac168-22d4-4a17-8dba-a0c7388e69a0","Type":"ContainerDied","Data":"e9c1399c07f22892c28bf7c65a3f5c2c19454327f5ea945d49afd6dbe9a71d2f"}
Jul 28 17:41:39.278341 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.278282    2561 scope.go:117] "RemoveContainer" containerID="63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"
Jul 28 17:41:39.287258 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.287244    2561 scope.go:117] "RemoveContainer" containerID="14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"
Jul 28 17:41:39.294522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.294507    2561 scope.go:117] "RemoveContainer" containerID="0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"
Jul 28 17:41:39.300254 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.300226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:41:39.302015 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.301999    2561 scope.go:117] "RemoveContainer" containerID="63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"
Jul 28 17:41:39.302692 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:41:39.302579    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879\": container with ID starting with 63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879 not found: ID does not exist" containerID="63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"
Jul 28 17:41:39.302692 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.302616    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879"} err="failed to get container status \"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879\": rpc error: code = NotFound desc = could not find container \"63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879\": container with ID starting with 63fd92f57af562daa8a9ed639ad0471fdf5c2e7a4309772fc6daecd79663b879 not found: ID does not exist"
Jul 28 17:41:39.302692 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.302638    2561 scope.go:117] "RemoveContainer" containerID="14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"
Jul 28 17:41:39.302925 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:41:39.302907    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396\": container with ID starting with 14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396 not found: ID does not exist" containerID="14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"
Jul 28 17:41:39.302989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.302935    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396"} err="failed to get container status \"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396\": rpc error: code = NotFound desc = could not find container \"14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396\": container with ID starting with 14c49b396cc369fef237a88cb5074aad2cfd57067e1e95d8c73033b1f62e3396 not found: ID does not exist"
Jul 28 17:41:39.302989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.302960    2561 scope.go:117] "RemoveContainer" containerID="0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"
Jul 28 17:41:39.303240 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:41:39.303220    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28\": container with ID starting with 0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28 not found: ID does not exist" containerID="0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"
Jul 28 17:41:39.303282 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.303246    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28"} err="failed to get container status \"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28\": rpc error: code = NotFound desc = could not find container \"0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28\": container with ID starting with 0afd0943aa87c9a5eb799628a2858a8a163afd8e237e98bb566f2623e5d08e28 not found: ID does not exist"
Jul 28 17:41:39.303975 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:39.303958    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5"]
Jul 28 17:41:40.203116 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:40.203082    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" path="/var/lib/kubelet/pods/ffcac168-22d4-4a17-8dba-a0c7388e69a0/volumes"
Jul 28 17:41:42.081697 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081663    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081951    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="storage-initializer"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081961    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="storage-initializer"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081973    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kube-rbac-proxy"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081979    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kube-rbac-proxy"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081990    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.081996    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.082048    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kube-rbac-proxy"
Jul 28 17:41:42.082081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.082057    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcac168-22d4-4a17-8dba-a0c7388e69a0" containerName="kserve-container"
Jul 28 17:41:42.087196 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.087176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.089805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.089781    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:41:42.089936 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.089783    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:41:42.089936 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.089810    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:41:42.089936 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.089867    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:41:42.090905 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.090888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:41:42.091018 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.090946    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:41:42.091083 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.091040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 17:41:42.095766 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.095736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:41:42.197659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.197624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.197838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.197666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.197838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.197685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sc6k\" (UniqueName: \"kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.197838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.197773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.197838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.197832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.298408 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.298639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.298639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sc6k\" (UniqueName: \"kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.298639 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.298810 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.299006 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.298980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.299124 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.299105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.299191 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.299134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.300923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.300901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.306210 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.306185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sc6k\" (UniqueName: \"kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.398955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.398853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:42.534620 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:42.534598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:41:42.536755 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:41:42.536730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662eeaec_2164_462f_bf0e_47210e9591ac.slice/crio-6fccb7f9604c0469d2e950fcd3e1e34809d61faa091d9a2ff3288c21d74f4293 WatchSource:0}: Error finding container 6fccb7f9604c0469d2e950fcd3e1e34809d61faa091d9a2ff3288c21d74f4293: Status 404 returned error can't find the container with id 6fccb7f9604c0469d2e950fcd3e1e34809d61faa091d9a2ff3288c21d74f4293
Jul 28 17:41:43.297037 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:43.297002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerStarted","Data":"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"}
Jul 28 17:41:43.297437 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:43.297071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerStarted","Data":"6fccb7f9604c0469d2e950fcd3e1e34809d61faa091d9a2ff3288c21d74f4293"}
Jul 28 17:41:52.335606 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:52.335573    2561 generic.go:356] "Generic (PLEG): container finished" podID="662eeaec-2164-462f-bf0e-47210e9591ac" containerID="e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f" exitCode=0
Jul 28 17:41:52.335992 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:52.335616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerDied","Data":"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"}
Jul 28 17:41:53.341807 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.341772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerStarted","Data":"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"}
Jul 28 17:41:53.342257 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.341832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerStarted","Data":"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"}
Jul 28 17:41:53.342257 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.342116    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:53.342257 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.342175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:53.343712 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.343680    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:41:53.361741 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:53.361686    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podStartSLOduration=2.319867403 podStartE2EDuration="11.361671934s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.041804531s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:53.359961815 +0000 UTC m=+1481.677400729" watchObservedRunningTime="2026-07-28 17:41:53.361671934 +0000 UTC m=+1481.679110821"
Jul 28 17:41:54.346481 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:54.346437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:41:59.351359 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:59.351300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:41:59.351939 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:41:59.351912    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:42:09.352506 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:09.352474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:42:17.317241 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:17.317211    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:42:17.317793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:17.317467    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container" containerID="cri-o://bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6" gracePeriod=30
Jul 28 17:42:17.317793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:17.317518    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kube-rbac-proxy" containerID="cri-o://464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6" gracePeriod=30
Jul 28 17:42:17.444374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:17.444342    2561 generic.go:356] "Generic (PLEG): container finished" podID="662eeaec-2164-462f-bf0e-47210e9591ac" containerID="464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6" exitCode=2
Jul 28 17:42:17.444526 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:17.444391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerDied","Data":"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"}
Jul 28 17:42:18.658332 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.658293    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:42:18.810795 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.810695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location\") pod \"662eeaec-2164-462f-bf0e-47210e9591ac\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") "
Jul 28 17:42:18.810795 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.810760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls\") pod \"662eeaec-2164-462f-bf0e-47210e9591ac\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") "
Jul 28 17:42:18.811039 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.810818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert\") pod \"662eeaec-2164-462f-bf0e-47210e9591ac\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") "
Jul 28 17:42:18.811039 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.810855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sc6k\" (UniqueName: \"kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k\") pod \"662eeaec-2164-462f-bf0e-47210e9591ac\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") "
Jul 28 17:42:18.811039 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.811012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config\") pod \"662eeaec-2164-462f-bf0e-47210e9591ac\" (UID: \"662eeaec-2164-462f-bf0e-47210e9591ac\") "
Jul 28 17:42:18.811292 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.811268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "662eeaec-2164-462f-bf0e-47210e9591ac" (UID: "662eeaec-2164-462f-bf0e-47210e9591ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:18.811393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.811369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "662eeaec-2164-462f-bf0e-47210e9591ac" (UID: "662eeaec-2164-462f-bf0e-47210e9591ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:18.811792 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.811771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "662eeaec-2164-462f-bf0e-47210e9591ac" (UID: "662eeaec-2164-462f-bf0e-47210e9591ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:18.813099 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.813071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "662eeaec-2164-462f-bf0e-47210e9591ac" (UID: "662eeaec-2164-462f-bf0e-47210e9591ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:18.813230 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.813182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k" (OuterVolumeSpecName: "kube-api-access-9sc6k") pod "662eeaec-2164-462f-bf0e-47210e9591ac" (UID: "662eeaec-2164-462f-bf0e-47210e9591ac"). InnerVolumeSpecName "kube-api-access-9sc6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:18.912053 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.912018    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/662eeaec-2164-462f-bf0e-47210e9591ac-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:42:18.912053 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.912045    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/662eeaec-2164-462f-bf0e-47210e9591ac-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:42:18.912053 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.912056    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:42:18.912053 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.912064    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sc6k\" (UniqueName: \"kubernetes.io/projected/662eeaec-2164-462f-bf0e-47210e9591ac-kube-api-access-9sc6k\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:42:18.912347 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:18.912072    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/662eeaec-2164-462f-bf0e-47210e9591ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:42:19.452849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.452810    2561 generic.go:356] "Generic (PLEG): container finished" podID="662eeaec-2164-462f-bf0e-47210e9591ac" containerID="bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6" exitCode=0
Jul 28 17:42:19.453021 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.452881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"
Jul 28 17:42:19.453021 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.452891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerDied","Data":"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"}
Jul 28 17:42:19.453021 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.452953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54" event={"ID":"662eeaec-2164-462f-bf0e-47210e9591ac","Type":"ContainerDied","Data":"6fccb7f9604c0469d2e950fcd3e1e34809d61faa091d9a2ff3288c21d74f4293"}
Jul 28 17:42:19.453021 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.452982    2561 scope.go:117] "RemoveContainer" containerID="464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"
Jul 28 17:42:19.461968 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.461953    2561 scope.go:117] "RemoveContainer" containerID="bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"
Jul 28 17:42:19.469337 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.469316    2561 scope.go:117] "RemoveContainer" containerID="e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"
Jul 28 17:42:19.475282 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.475255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:42:19.476810 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.476798    2561 scope.go:117] "RemoveContainer" containerID="464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"
Jul 28 17:42:19.477382 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:42:19.477105    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6\": container with ID starting with 464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6 not found: ID does not exist" containerID="464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"
Jul 28 17:42:19.477382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.477162    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6"} err="failed to get container status \"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6\": rpc error: code = NotFound desc = could not find container \"464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6\": container with ID starting with 464a249e94e3bcbb3157de80cffc2c28480e8532998790802d5507321c07e9b6 not found: ID does not exist"
Jul 28 17:42:19.477382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.477186    2561 scope.go:117] "RemoveContainer" containerID="bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"
Jul 28 17:42:19.477600 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:42:19.477510    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6\": container with ID starting with bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6 not found: ID does not exist" containerID="bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"
Jul 28 17:42:19.477600 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.477542    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6"} err="failed to get container status \"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6\": rpc error: code = NotFound desc = could not find container \"bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6\": container with ID starting with bb22d41825981955472cc9106a6e55a8995c9ad258c5baf80b732e749e0dbbe6 not found: ID does not exist"
Jul 28 17:42:19.477600 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.477565    2561 scope.go:117] "RemoveContainer" containerID="e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"
Jul 28 17:42:19.477812 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:42:19.477794    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f\": container with ID starting with e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f not found: ID does not exist" containerID="e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"
Jul 28 17:42:19.477855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.477817    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f"} err="failed to get container status \"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f\": rpc error: code = NotFound desc = could not find container \"e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f\": container with ID starting with e4f3aae3d18cbf074cf553a3427e3c124acbabb1846522649eb9b42cb929f66f not found: ID does not exist"
Jul 28 17:42:19.478849 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:19.478832    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54"]
Jul 28 17:42:20.204109 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:20.204076    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" path="/var/lib/kubelet/pods/662eeaec-2164-462f-bf0e-47210e9591ac/volumes"
Jul 28 17:42:22.523253 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523223    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523557    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="storage-initializer"
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523570    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="storage-initializer"
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523579    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container"
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container"
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523604    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kube-rbac-proxy"
Jul 28 17:42:22.523644 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523610    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kube-rbac-proxy"
Jul 28 17:42:22.523835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523656    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kube-rbac-proxy"
Jul 28 17:42:22.523835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.523664    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="662eeaec-2164-462f-bf0e-47210e9591ac" containerName="kserve-container"
Jul 28 17:42:22.529621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.529603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.532455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:42:22.532455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:42:22.532455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:42:22.532455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532399    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:42:22.532751 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:42:22.532751 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.532733    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:42:22.533367 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.533334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:42:22.538406 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.538386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:42:22.563751 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.563725    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:42:22.567415 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.567390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.569715 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.569697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:42:22.579661 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.579639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:42:22.642143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.642108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9mj\" (UniqueName: \"kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.642336 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.642165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.642336 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.642186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.642336 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.642237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.642336 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.642281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.743632 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.743838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.743838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.743838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.743838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55mp\" (UniqueName: \"kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.743838 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9mj\" (UniqueName: \"kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.744098 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.744098 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.744098 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.743931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.744098 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.744015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.744342 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.744298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.744496 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.744475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.744727 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.744691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.746328 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.746290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.752266 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.752241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9mj\" (UniqueName: \"kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj\") pod \"isvc-canary-create-predictor-cb85cf55c-7jhbv\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.841208 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.841125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55mp\" (UniqueName: \"kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.844561 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:42:22.844534    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:42:22.844884 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:42:22.844603    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls podName:b70f993f-7a58-48da-b0c9-3dda667e4cf3 nodeName:}" failed. No retries permitted until 2026-07-28 17:42:23.344585119 +0000 UTC m=+1511.662023985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls") pod "isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:42:22.844884 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.844840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.845071 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.845051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.845224 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.845201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.855238 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.855213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55mp\" (UniqueName: \"kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:22.976207 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:22.976185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:42:22.978634 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:42:22.978608    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d9dc5_1044_4304_8986_12ad478214a1.slice/crio-4afaa43d0f24c99c8132cf4777fca0a643f81048dfbe3ac8c59bb3cd1e222051 WatchSource:0}: Error finding container 4afaa43d0f24c99c8132cf4777fca0a643f81048dfbe3ac8c59bb3cd1e222051: Status 404 returned error can't find the container with id 4afaa43d0f24c99c8132cf4777fca0a643f81048dfbe3ac8c59bb3cd1e222051
Jul 28 17:42:23.348962 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.348926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:23.351317 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.351277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:23.469858 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.469821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerStarted","Data":"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"}
Jul 28 17:42:23.470065 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.469878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerStarted","Data":"4afaa43d0f24c99c8132cf4777fca0a643f81048dfbe3ac8c59bb3cd1e222051"}
Jul 28 17:42:23.477413 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.477385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:23.606851 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:23.606821    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:42:23.608984 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:42:23.608958    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70f993f_7a58_48da_b0c9_3dda667e4cf3.slice/crio-2bca09eee60e7e07bde4fad9021d6422df9431f76f9ba77b84f73ee40d8e8b72 WatchSource:0}: Error finding container 2bca09eee60e7e07bde4fad9021d6422df9431f76f9ba77b84f73ee40d8e8b72: Status 404 returned error can't find the container with id 2bca09eee60e7e07bde4fad9021d6422df9431f76f9ba77b84f73ee40d8e8b72
Jul 28 17:42:24.475376 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:24.475333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerStarted","Data":"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"}
Jul 28 17:42:24.475561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:24.475417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerStarted","Data":"2bca09eee60e7e07bde4fad9021d6422df9431f76f9ba77b84f73ee40d8e8b72"}
Jul 28 17:42:27.488983 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:27.488950    2561 generic.go:356] "Generic (PLEG): container finished" podID="e94d9dc5-1044-4304-8986-12ad478214a1" containerID="662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a" exitCode=0
Jul 28 17:42:27.489382 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:27.489024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerDied","Data":"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"}
Jul 28 17:42:27.490666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:27.490645    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerID="00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813" exitCode=0
Jul 28 17:42:27.490756 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:27.490677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerDied","Data":"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"}
Jul 28 17:42:28.496784 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.496744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerStarted","Data":"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"}
Jul 28 17:42:28.497248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.496815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerStarted","Data":"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"}
Jul 28 17:42:28.497248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.497146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:28.497248 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.497212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:28.498794 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.498768    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:42:28.498915 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.498828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerStarted","Data":"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"}
Jul 28 17:42:28.498915 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.498869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerStarted","Data":"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"}
Jul 28 17:42:28.499189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.499172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:28.499232 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.499218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:28.500192 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.500171    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:42:28.517694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.517652    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podStartSLOduration=2.816839865 podStartE2EDuration="6.517636045s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70079618s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:28.516337206 +0000 UTC m=+1516.833776095" watchObservedRunningTime="2026-07-28 17:42:28.517636045 +0000 UTC m=+1516.835074938"
Jul 28 17:42:28.537023 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:28.536980    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podStartSLOduration=2.730283586 podStartE2EDuration="6.536965423s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806681837s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:28.535480549 +0000 UTC m=+1516.852919438" watchObservedRunningTime="2026-07-28 17:42:28.536965423 +0000 UTC m=+1516.854404728"
Jul 28 17:42:29.503146 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:29.503103    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:42:29.503684 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:29.503234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:42:34.508073 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:34.507994    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:42:34.508533 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:34.508410    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:42:34.508533 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:34.508510    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:42:34.508925 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:34.508901    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:42:44.509034 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:44.508990    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:42:44.509559 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:44.508991    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:42:54.509457 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:54.509420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:42:54.509844 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:42:54.509427    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:04.509446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:04.509397    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:04.509446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:04.509421    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:43:14.509205 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:14.509163    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:43:14.509640 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:14.509165    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:24.508841 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:24.508805    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:43:24.509295 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:24.508804    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:34.509158 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:34.509119    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:43:34.511562 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:34.509132    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:35.199149 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:35.199104    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:39.198692 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:39.198653    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 17:43:45.199460 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:45.199434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:43:49.199207 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:49.199178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:43:52.655045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.655009    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:43:52.655532 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.655264    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container" containerID="cri-o://5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec" gracePeriod=30
Jul 28 17:43:52.655532 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.655342    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kube-rbac-proxy" containerID="cri-o://4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb" gracePeriod=30
Jul 28 17:43:52.659330 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.659238    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:43:52.659595 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.659548    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" containerID="cri-o://1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0" gracePeriod=30
Jul 28 17:43:52.659717 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.659568    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kube-rbac-proxy" containerID="cri-o://f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4" gracePeriod=30
Jul 28 17:43:52.846284 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.846250    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerID="4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb" exitCode=2
Jul 28 17:43:52.846474 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.846335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerDied","Data":"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"}
Jul 28 17:43:52.848395 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.848373    2561 generic.go:356] "Generic (PLEG): container finished" podID="e94d9dc5-1044-4304-8986-12ad478214a1" containerID="f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4" exitCode=2
Jul 28 17:43:52.848498 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:52.848425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerDied","Data":"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"}
Jul 28 17:43:54.503658 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:54.503615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 17:43:54.504045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:54.503614    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 28 17:43:55.198885 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:55.198843    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:43:57.392780 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.392757    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:43:57.546747 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.546659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location\") pod \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") "
Jul 28 17:43:57.546747 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.546719    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert\") pod \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") "
Jul 28 17:43:57.546989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.546752    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config\") pod \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") "
Jul 28 17:43:57.546989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.546773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s55mp\" (UniqueName: \"kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp\") pod \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") "
Jul 28 17:43:57.546989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.546796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") pod \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\" (UID: \"b70f993f-7a58-48da-b0c9-3dda667e4cf3\") "
Jul 28 17:43:57.547163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.547039    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b70f993f-7a58-48da-b0c9-3dda667e4cf3" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:57.547211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.547152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b70f993f-7a58-48da-b0c9-3dda667e4cf3" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:57.547211 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.547159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b70f993f-7a58-48da-b0c9-3dda667e4cf3" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:57.548875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.548855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b70f993f-7a58-48da-b0c9-3dda667e4cf3" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:57.548951 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.548872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp" (OuterVolumeSpecName: "kube-api-access-s55mp") pod "b70f993f-7a58-48da-b0c9-3dda667e4cf3" (UID: "b70f993f-7a58-48da-b0c9-3dda667e4cf3"). InnerVolumeSpecName "kube-api-access-s55mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:57.648059 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.648025    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.648059 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.648052    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.648059 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.648062    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.648273 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.648070    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s55mp\" (UniqueName: \"kubernetes.io/projected/b70f993f-7a58-48da-b0c9-3dda667e4cf3-kube-api-access-s55mp\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.648273 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.648079    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70f993f-7a58-48da-b0c9-3dda667e4cf3-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.792172 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.792148    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:43:57.818501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818418    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:43:57.818740 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818727    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="storage-initializer"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818741    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="storage-initializer"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818750    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kube-rbac-proxy"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kube-rbac-proxy"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818769    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818775    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818785    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="storage-initializer"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818791    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="storage-initializer"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818797    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818802    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818808    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kube-rbac-proxy"
Jul 28 17:43:57.818809 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818813    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kube-rbac-proxy"
Jul 28 17:43:57.819409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818863    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kserve-container"
Jul 28 17:43:57.819409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818871    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerName="kube-rbac-proxy"
Jul 28 17:43:57.819409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818877    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kserve-container"
Jul 28 17:43:57.819409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.818885    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" containerName="kube-rbac-proxy"
Jul 28 17:43:57.821839 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.821819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.824100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.824082    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:43:57.824100 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.824089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:43:57.830633 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.830613    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:43:57.834321 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.834285    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:43:57.834445 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.834426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.836702 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.836681    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:43:57.844070 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.844050    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:43:57.873684 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.873653    2561 generic.go:356] "Generic (PLEG): container finished" podID="e94d9dc5-1044-4304-8986-12ad478214a1" containerID="1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0" exitCode=0
Jul 28 17:43:57.873826 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.873707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"
Jul 28 17:43:57.873826 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.873737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerDied","Data":"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"}
Jul 28 17:43:57.873826 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.873791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv" event={"ID":"e94d9dc5-1044-4304-8986-12ad478214a1","Type":"ContainerDied","Data":"4afaa43d0f24c99c8132cf4777fca0a643f81048dfbe3ac8c59bb3cd1e222051"}
Jul 28 17:43:57.873980 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.873826    2561 scope.go:117] "RemoveContainer" containerID="f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"
Jul 28 17:43:57.875543 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.875524    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" containerID="5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec" exitCode=0
Jul 28 17:43:57.875643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.875577    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"
Jul 28 17:43:57.875643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.875578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerDied","Data":"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"}
Jul 28 17:43:57.875643 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.875626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc" event={"ID":"b70f993f-7a58-48da-b0c9-3dda667e4cf3","Type":"ContainerDied","Data":"2bca09eee60e7e07bde4fad9021d6422df9431f76f9ba77b84f73ee40d8e8b72"}
Jul 28 17:43:57.882587 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.882554    2561 scope.go:117] "RemoveContainer" containerID="1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"
Jul 28 17:43:57.889744 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.889725    2561 scope.go:117] "RemoveContainer" containerID="662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"
Jul 28 17:43:57.896847 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.896830    2561 scope.go:117] "RemoveContainer" containerID="f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"
Jul 28 17:43:57.897069 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.897054    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4\": container with ID starting with f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4 not found: ID does not exist" containerID="f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"
Jul 28 17:43:57.897119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897076    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4"} err="failed to get container status \"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4\": rpc error: code = NotFound desc = could not find container \"f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4\": container with ID starting with f08befbe44478fa0ebe14ed9cd02617fbd368b16b6570054a88d65be3b5568e4 not found: ID does not exist"
Jul 28 17:43:57.897119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897090    2561 scope.go:117] "RemoveContainer" containerID="1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"
Jul 28 17:43:57.897332 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.897295    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0\": container with ID starting with 1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0 not found: ID does not exist" containerID="1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"
Jul 28 17:43:57.897403 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897344    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0"} err="failed to get container status \"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0\": rpc error: code = NotFound desc = could not find container \"1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0\": container with ID starting with 1c8e712432da53e72b8854571c0ea3f5ff50d48e5bad5351c0bc03aa424715e0 not found: ID does not exist"
Jul 28 17:43:57.897403 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897367    2561 scope.go:117] "RemoveContainer" containerID="662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"
Jul 28 17:43:57.897668 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.897652    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a\": container with ID starting with 662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a not found: ID does not exist" containerID="662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"
Jul 28 17:43:57.897727 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897681    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a"} err="failed to get container status \"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a\": rpc error: code = NotFound desc = could not find container \"662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a\": container with ID starting with 662940c61e2ab7dce7ffbc2bb093c541b3bb185a42e29f39fab6638fe1a0396a not found: ID does not exist"
Jul 28 17:43:57.897727 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.897694    2561 scope.go:117] "RemoveContainer" containerID="4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"
Jul 28 17:43:57.901189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.901111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:43:57.903221 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.903186    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc"]
Jul 28 17:43:57.907289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.907269    2561 scope.go:117] "RemoveContainer" containerID="5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"
Jul 28 17:43:57.914781 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.914747    2561 scope.go:117] "RemoveContainer" containerID="00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"
Jul 28 17:43:57.921680 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.921660    2561 scope.go:117] "RemoveContainer" containerID="4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"
Jul 28 17:43:57.921890 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.921873    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb\": container with ID starting with 4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb not found: ID does not exist" containerID="4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"
Jul 28 17:43:57.921940 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.921899    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb"} err="failed to get container status \"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb\": rpc error: code = NotFound desc = could not find container \"4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb\": container with ID starting with 4e5e374ddd87a0df0e0edb17688765e9b49696d7a98851a6e3a7f0573e8027bb not found: ID does not exist"
Jul 28 17:43:57.921940 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.921917    2561 scope.go:117] "RemoveContainer" containerID="5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"
Jul 28 17:43:57.922161 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.922145    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec\": container with ID starting with 5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec not found: ID does not exist" containerID="5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"
Jul 28 17:43:57.922218 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.922165    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec"} err="failed to get container status \"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec\": rpc error: code = NotFound desc = could not find container \"5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec\": container with ID starting with 5f736459f069f15d58418ef8144c033cd8b68f923c04a1b8a04d174bea975bec not found: ID does not exist"
Jul 28 17:43:57.922218 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.922178    2561 scope.go:117] "RemoveContainer" containerID="00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"
Jul 28 17:43:57.922365 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:57.922347    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813\": container with ID starting with 00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813 not found: ID does not exist" containerID="00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"
Jul 28 17:43:57.922408 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.922370    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813"} err="failed to get container status \"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813\": rpc error: code = NotFound desc = could not find container \"00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813\": container with ID starting with 00a1956cb63d166547f8250cee8c732a58c45c05aa16b07c9edc7456bc78a813 not found: ID does not exist"
Jul 28 17:43:57.951146 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951127    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg9mj\" (UniqueName: \"kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj\") pod \"e94d9dc5-1044-4304-8986-12ad478214a1\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") "
Jul 28 17:43:57.951246 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951199    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location\") pod \"e94d9dc5-1044-4304-8986-12ad478214a1\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") "
Jul 28 17:43:57.951328 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert\") pod \"e94d9dc5-1044-4304-8986-12ad478214a1\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") "
Jul 28 17:43:57.951384 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls\") pod \"e94d9dc5-1044-4304-8986-12ad478214a1\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") "
Jul 28 17:43:57.951435 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config\") pod \"e94d9dc5-1044-4304-8986-12ad478214a1\" (UID: \"e94d9dc5-1044-4304-8986-12ad478214a1\") "
Jul 28 17:43:57.951514 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951493    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e94d9dc5-1044-4304-8986-12ad478214a1" (UID: "e94d9dc5-1044-4304-8986-12ad478214a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:57.951563 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5s6\" (UniqueName: \"kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.951615 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8px\" (UniqueName: \"kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.951615 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e94d9dc5-1044-4304-8986-12ad478214a1" (UID: "e94d9dc5-1044-4304-8986-12ad478214a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:57.951737 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.951796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.951796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e94d9dc5-1044-4304-8986-12ad478214a1" (UID: "e94d9dc5-1044-4304-8986-12ad478214a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:57.951876 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.951876 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.951955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.951955 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:57.952055 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.951965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.952055 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.952012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:57.952143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.952066    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e94d9dc5-1044-4304-8986-12ad478214a1-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.952143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.952084    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.952143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.952100    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e94d9dc5-1044-4304-8986-12ad478214a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:57.953188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.953169    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj" (OuterVolumeSpecName: "kube-api-access-hg9mj") pod "e94d9dc5-1044-4304-8986-12ad478214a1" (UID: "e94d9dc5-1044-4304-8986-12ad478214a1"). InnerVolumeSpecName "kube-api-access-hg9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:57.953252 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:57.953230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e94d9dc5-1044-4304-8986-12ad478214a1" (UID: "e94d9dc5-1044-4304-8986-12ad478214a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:58.053205 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.053205 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5s6\" (UniqueName: \"kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8px\" (UniqueName: \"kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:58.053484    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053493    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e94d9dc5-1044-4304-8986-12ad478214a1-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:58.053501 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053508    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg9mj\" (UniqueName: \"kubernetes.io/projected/e94d9dc5-1044-4304-8986-12ad478214a1-kube-api-access-hg9mj\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:43:58.054032 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:43:58.053551    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls podName:ba39c2e9-42d4-4b2c-819d-d54c585ca9e8 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:58.553527911 +0000 UTC m=+1606.870966793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls") pod "isvc-canary-promote-v2-predictor-86766c4958-mxmmx" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:43:58.054032 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.054032 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.054032 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.053994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.054171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.054034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.054171 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.054086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.054257 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.054238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.055910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.055893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.061863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.061840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8px\" (UniqueName: \"kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.061986 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.061967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5s6\" (UniqueName: \"kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6\") pod \"isvc-canary-promote-predictor-d9759ffd5-m74zc\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.132679 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.132604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:43:58.207161 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.207127    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b70f993f-7a58-48da-b0c9-3dda667e4cf3" path="/var/lib/kubelet/pods/b70f993f-7a58-48da-b0c9-3dda667e4cf3/volumes"
Jul 28 17:43:58.207523 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.207508    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:43:58.207575 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.207539    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv"]
Jul 28 17:43:58.270048 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.270011    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:43:58.273349 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:43:58.273323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8377fc0e_a6ff_4f65_8811_d026878fd8c1.slice/crio-f4fb9b4b866db184472883c048ff5a840017c9981baabf239839211d120254f2 WatchSource:0}: Error finding container f4fb9b4b866db184472883c048ff5a840017c9981baabf239839211d120254f2: Status 404 returned error can't find the container with id f4fb9b4b866db184472883c048ff5a840017c9981baabf239839211d120254f2
Jul 28 17:43:58.559236 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.559196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.561759 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.561732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-mxmmx\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.743834 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.743799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:43:58.876433 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.876407    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:43:58.879089 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:43:58.879064    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba39c2e9_42d4_4b2c_819d_d54c585ca9e8.slice/crio-91012f862d1880ffbe85a7fa1b4d355bc14a3161dc6744a196db7626e39b75d8 WatchSource:0}: Error finding container 91012f862d1880ffbe85a7fa1b4d355bc14a3161dc6744a196db7626e39b75d8: Status 404 returned error can't find the container with id 91012f862d1880ffbe85a7fa1b4d355bc14a3161dc6744a196db7626e39b75d8
Jul 28 17:43:58.883358 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.883334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerStarted","Data":"a0df9bcc06b6642065626fa8099e686c7cfab4bd6edc6827a3fba0b3668bb326"}
Jul 28 17:43:58.883445 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:58.883384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerStarted","Data":"f4fb9b4b866db184472883c048ff5a840017c9981baabf239839211d120254f2"}
Jul 28 17:43:59.896380 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:59.896338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerStarted","Data":"b062e4f78d9e23bd43d1e4a45e0510996cb9bc8927cbbaac86959670d97a7d76"}
Jul 28 17:43:59.896765 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:43:59.896421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerStarted","Data":"91012f862d1880ffbe85a7fa1b4d355bc14a3161dc6744a196db7626e39b75d8"}
Jul 28 17:44:00.203383 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:00.203352    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94d9dc5-1044-4304-8986-12ad478214a1" path="/var/lib/kubelet/pods/e94d9dc5-1044-4304-8986-12ad478214a1/volumes"
Jul 28 17:44:02.910708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:02.910679    2561 generic.go:356] "Generic (PLEG): container finished" podID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerID="a0df9bcc06b6642065626fa8099e686c7cfab4bd6edc6827a3fba0b3668bb326" exitCode=0
Jul 28 17:44:02.911107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:02.910755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerDied","Data":"a0df9bcc06b6642065626fa8099e686c7cfab4bd6edc6827a3fba0b3668bb326"}
Jul 28 17:44:02.912356 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:02.912336    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerID="b062e4f78d9e23bd43d1e4a45e0510996cb9bc8927cbbaac86959670d97a7d76" exitCode=0
Jul 28 17:44:02.912456 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:02.912374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerDied","Data":"b062e4f78d9e23bd43d1e4a45e0510996cb9bc8927cbbaac86959670d97a7d76"}
Jul 28 17:44:03.924910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.919943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerStarted","Data":"3b66bd3a41b6b0675cb48fe7032484bab9073691b53e75c558e3460d2fd818ca"}
Jul 28 17:44:03.924910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.920071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerStarted","Data":"898518d47f913cc8136bf335148a6f9e6b82afc572845146b2f47c5988a4cb5e"}
Jul 28 17:44:03.924910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.920897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:44:03.924910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.920958    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:44:03.924910 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.921758    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:03.927086 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.927060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerStarted","Data":"be16a1096bc39d78afd10c91b7a10854f08bf4d6802039d10de66695182e6e7b"}
Jul 28 17:44:03.927216 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.927118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerStarted","Data":"101beca2ac8f250a8eafc9b53bd7ca210de15b6edb9a9e44a97fafaf610ca27b"}
Jul 28 17:44:03.927416 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.927390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:44:03.939896 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.939851    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podStartSLOduration=3.192943582 podStartE2EDuration="6.939838427s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746894845s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:03.938931871 +0000 UTC m=+1612.256370781" watchObservedRunningTime="2026-07-28 17:44:03.939838427 +0000 UTC m=+1612.257277313"
Jul 28 17:44:03.959935 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:03.959894    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podStartSLOduration=3.324859347 podStartE2EDuration="6.959879925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635020578s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:03.958738901 +0000 UTC m=+1612.276177789" watchObservedRunningTime="2026-07-28 17:44:03.959879925 +0000 UTC m=+1612.277318811"
Jul 28 17:44:04.932232 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:04.932186    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:04.932709 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:04.932337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:44:04.933392 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:04.933370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:05.937398 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:05.937358    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:09.936488 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:09.936460    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:44:09.937000 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:09.936973    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:10.941986 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:10.941957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:44:10.942548 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:10.942521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:19.937826 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:19.937787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:20.943045 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:20.943002    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:29.937541 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:29.937502    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:30.943223 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:30.943182    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:39.937686 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:39.937641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:40.942559 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:40.942523    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:49.937234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:49.937195    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:44:50.943355 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:50.943299    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:44:59.937750 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:44:59.937709    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:45:00.942570 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:00.942530    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:45:09.936923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:09.936884    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:45:10.942755 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:10.942718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:45:14.202859 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:14.202828    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:45:20.943050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:20.943023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:45:22.875355 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:22.875322    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:45:22.875922 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:22.875568    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" containerID="cri-o://898518d47f913cc8136bf335148a6f9e6b82afc572845146b2f47c5988a4cb5e" gracePeriod=30
Jul 28 17:45:22.875922 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:22.875590    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kube-rbac-proxy" containerID="cri-o://3b66bd3a41b6b0675cb48fe7032484bab9073691b53e75c558e3460d2fd818ca" gracePeriod=30
Jul 28 17:45:23.250678 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:23.250643    2561 generic.go:356] "Generic (PLEG): container finished" podID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerID="3b66bd3a41b6b0675cb48fe7032484bab9073691b53e75c558e3460d2fd818ca" exitCode=2
Jul 28 17:45:23.250850 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:23.250716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerDied","Data":"3b66bd3a41b6b0675cb48fe7032484bab9073691b53e75c558e3460d2fd818ca"}
Jul 28 17:45:24.198469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:24.198426    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:45:24.933253 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:24.933208    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:45:27.268907 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.268876    2561 generic.go:356] "Generic (PLEG): container finished" podID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerID="898518d47f913cc8136bf335148a6f9e6b82afc572845146b2f47c5988a4cb5e" exitCode=0
Jul 28 17:45:27.269239 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.268954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerDied","Data":"898518d47f913cc8136bf335148a6f9e6b82afc572845146b2f47c5988a4cb5e"}
Jul 28 17:45:27.314399 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.314376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:45:27.396299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location\") pod \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") "
Jul 28 17:45:27.396299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert\") pod \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") "
Jul 28 17:45:27.396299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls\") pod \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") "
Jul 28 17:45:27.396579 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf5s6\" (UniqueName: \"kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6\") pod \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") "
Jul 28 17:45:27.396579 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396443    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config\") pod \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\" (UID: \"8377fc0e-a6ff-4f65-8811-d026878fd8c1\") "
Jul 28 17:45:27.396689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8377fc0e-a6ff-4f65-8811-d026878fd8c1" (UID: "8377fc0e-a6ff-4f65-8811-d026878fd8c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:27.396689 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8377fc0e-a6ff-4f65-8811-d026878fd8c1" (UID: "8377fc0e-a6ff-4f65-8811-d026878fd8c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:27.396804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396785    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:45:27.396847 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.396817    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8377fc0e-a6ff-4f65-8811-d026878fd8c1" (UID: "8377fc0e-a6ff-4f65-8811-d026878fd8c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:27.398352 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.398333    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8377fc0e-a6ff-4f65-8811-d026878fd8c1" (UID: "8377fc0e-a6ff-4f65-8811-d026878fd8c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:27.398441 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.398380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6" (OuterVolumeSpecName: "kube-api-access-tf5s6") pod "8377fc0e-a6ff-4f65-8811-d026878fd8c1" (UID: "8377fc0e-a6ff-4f65-8811-d026878fd8c1"). InnerVolumeSpecName "kube-api-access-tf5s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:27.497477 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.497450    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:45:27.497477 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.497470    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8377fc0e-a6ff-4f65-8811-d026878fd8c1-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:45:27.497477 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.497481    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8377fc0e-a6ff-4f65-8811-d026878fd8c1-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:45:27.497665 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:27.497489    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5s6\" (UniqueName: \"kubernetes.io/projected/8377fc0e-a6ff-4f65-8811-d026878fd8c1-kube-api-access-tf5s6\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.274850 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.274817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc" event={"ID":"8377fc0e-a6ff-4f65-8811-d026878fd8c1","Type":"ContainerDied","Data":"f4fb9b4b866db184472883c048ff5a840017c9981baabf239839211d120254f2"}
Jul 28 17:45:28.275188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.274883    2561 scope.go:117] "RemoveContainer" containerID="3b66bd3a41b6b0675cb48fe7032484bab9073691b53e75c558e3460d2fd818ca"
Jul 28 17:45:28.275188 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.274833    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"
Jul 28 17:45:28.282880 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.282863    2561 scope.go:117] "RemoveContainer" containerID="898518d47f913cc8136bf335148a6f9e6b82afc572845146b2f47c5988a4cb5e"
Jul 28 17:45:28.291845 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.291827    2561 scope.go:117] "RemoveContainer" containerID="a0df9bcc06b6642065626fa8099e686c7cfab4bd6edc6827a3fba0b3668bb326"
Jul 28 17:45:28.293962 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.293942    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:45:28.297077 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:28.297058    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc"]
Jul 28 17:45:30.203730 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:45:30.203698    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" path="/var/lib/kubelet/pods/8377fc0e-a6ff-4f65-8811-d026878fd8c1/volumes"
Jul 28 17:47:13.171649 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:13.171616    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:47:13.172177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:13.171916    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container" containerID="cri-o://101beca2ac8f250a8eafc9b53bd7ca210de15b6edb9a9e44a97fafaf610ca27b" gracePeriod=30
Jul 28 17:47:13.172177 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:13.171944    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kube-rbac-proxy" containerID="cri-o://be16a1096bc39d78afd10c91b7a10854f08bf4d6802039d10de66695182e6e7b" gracePeriod=30
Jul 28 17:47:13.678853 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:13.678822    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerID="be16a1096bc39d78afd10c91b7a10854f08bf4d6802039d10de66695182e6e7b" exitCode=2
Jul 28 17:47:13.679016 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:13.678879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerDied","Data":"be16a1096bc39d78afd10c91b7a10854f08bf4d6802039d10de66695182e6e7b"}
Jul 28 17:47:15.937853 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:15.937811    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 28 17:47:17.695162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.695128    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerID="101beca2ac8f250a8eafc9b53bd7ca210de15b6edb9a9e44a97fafaf610ca27b" exitCode=0
Jul 28 17:47:17.695625 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.695198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerDied","Data":"101beca2ac8f250a8eafc9b53bd7ca210de15b6edb9a9e44a97fafaf610ca27b"}
Jul 28 17:47:17.717299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.717280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:47:17.774622 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn8px\" (UniqueName: \"kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px\") pod \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") "
Jul 28 17:47:17.774622 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert\") pod \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") "
Jul 28 17:47:17.774622 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config\") pod \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") "
Jul 28 17:47:17.774869 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") pod \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") "
Jul 28 17:47:17.774869 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location\") pod \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\" (UID: \"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8\") "
Jul 28 17:47:17.775016 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.774990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:17.775060 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.775036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:17.775112 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.775092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:17.776659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.776637    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px" (OuterVolumeSpecName: "kube-api-access-rn8px") pod "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8"). InnerVolumeSpecName "kube-api-access-rn8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:17.776659 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.776653    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" (UID: "ba39c2e9-42d4-4b2c-819d-d54c585ca9e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:17.875275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.875249    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8px\" (UniqueName: \"kubernetes.io/projected/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-api-access-rn8px\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:47:17.875275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.875270    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:47:17.875275 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.875280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:47:17.875475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.875289    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:47:17.875475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:17.875299    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.368319 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368285    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:47:18.368605 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368594    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kube-rbac-proxy"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368608    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kube-rbac-proxy"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368617    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368622    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kube-rbac-proxy"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kube-rbac-proxy"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="storage-initializer"
Jul 28 17:47:18.368648 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368645    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="storage-initializer"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368656    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="storage-initializer"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368661    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="storage-initializer"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368677    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kserve-container"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368733    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kserve-container"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368740    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" containerName="kube-rbac-proxy"
Jul 28 17:47:18.368854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.368747    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8377fc0e-a6ff-4f65-8811-d026878fd8c1" containerName="kube-rbac-proxy"
Jul 28 17:47:18.371865 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.371850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.375691 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.375672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:47:18.376132 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.376114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:47:18.385163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.385143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:47:18.479996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.479917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.479996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.479951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.479996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.479980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.480189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.480052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zw6\" (UniqueName: \"kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.480189 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.480088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.580574 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.580538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zw6\" (UniqueName: \"kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.580574 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.580577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.580804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.580631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.580804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.580654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.580804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.580694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.581026 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.581007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.581294 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.581263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.581417 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.581382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.583124 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.583108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.588578 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.588556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zw6\" (UniqueName: \"kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-klkls\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.681682 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.681656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:18.701508 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.701487    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"
Jul 28 17:47:18.701881 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.701477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx" event={"ID":"ba39c2e9-42d4-4b2c-819d-d54c585ca9e8","Type":"ContainerDied","Data":"91012f862d1880ffbe85a7fa1b4d355bc14a3161dc6744a196db7626e39b75d8"}
Jul 28 17:47:18.701881 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.701632    2561 scope.go:117] "RemoveContainer" containerID="be16a1096bc39d78afd10c91b7a10854f08bf4d6802039d10de66695182e6e7b"
Jul 28 17:47:18.710116 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.710096    2561 scope.go:117] "RemoveContainer" containerID="101beca2ac8f250a8eafc9b53bd7ca210de15b6edb9a9e44a97fafaf610ca27b"
Jul 28 17:47:18.717687 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.717667    2561 scope.go:117] "RemoveContainer" containerID="b062e4f78d9e23bd43d1e4a45e0510996cb9bc8927cbbaac86959670d97a7d76"
Jul 28 17:47:18.720335 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.720301    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:47:18.724414 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.724393    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx"]
Jul 28 17:47:18.815599 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:18.815574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:47:18.818140 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:47:18.818117    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67e4eb_345b_48a8_ac2f_89c5c2afce38.slice/crio-5eb61cfd4f2b0f929589a83d2b61447a6006dcb0090fd51541cbb063fbc31c86 WatchSource:0}: Error finding container 5eb61cfd4f2b0f929589a83d2b61447a6006dcb0090fd51541cbb063fbc31c86: Status 404 returned error can't find the container with id 5eb61cfd4f2b0f929589a83d2b61447a6006dcb0090fd51541cbb063fbc31c86
Jul 28 17:47:19.706919 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:19.706883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerStarted","Data":"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"}
Jul 28 17:47:19.707410 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:19.706956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerStarted","Data":"5eb61cfd4f2b0f929589a83d2b61447a6006dcb0090fd51541cbb063fbc31c86"}
Jul 28 17:47:20.202637 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:20.202604    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba39c2e9-42d4-4b2c-819d-d54c585ca9e8" path="/var/lib/kubelet/pods/ba39c2e9-42d4-4b2c-819d-d54c585ca9e8/volumes"
Jul 28 17:47:22.721286 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:22.721249    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerID="29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f" exitCode=0
Jul 28 17:47:22.721640 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:22.721342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerDied","Data":"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"}
Jul 28 17:47:23.726927 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.726894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerStarted","Data":"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"}
Jul 28 17:47:23.727367 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.726953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerStarted","Data":"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"}
Jul 28 17:47:23.727367 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.727226    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:23.727507 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.727384    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:23.728757 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.728734    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:47:23.747262 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:23.747219    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podStartSLOduration=1.9700375079999999 podStartE2EDuration="5.747205982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777168474s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:23.745103101 +0000 UTC m=+1812.062541988" watchObservedRunningTime="2026-07-28 17:47:23.747205982 +0000 UTC m=+1812.064644916"
Jul 28 17:47:24.730906 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:24.730863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:47:29.735535 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:29.735507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:47:29.736154 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:29.736127    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:47:39.737012 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:39.736969    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:47:49.736148 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:49.736105    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:47:59.736268 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:47:59.736229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:48:09.736282 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:09.736244    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:48:19.736828 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:19.736790    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:48:29.736321 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:29.736267    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:48:39.737409 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:39.737379    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:48:43.342708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.342677    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:48:43.343070 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.342973    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container" containerID="cri-o://f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481" gracePeriod=30
Jul 28 17:48:43.343070 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.343019    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kube-rbac-proxy" containerID="cri-o://f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f" gracePeriod=30
Jul 28 17:48:43.383803 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.383767    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:48:43.390068 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.390044    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.393122 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.393100    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:48:43.398331 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.398277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:48:43.474559 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.474528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.474709 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.474596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9dd\" (UniqueName: \"kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.474709 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.474618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.474785 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.474704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.474785 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.474731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.575663 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.575631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9dd\" (UniqueName: \"kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.575663 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.575664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.575855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.575703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.575855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.575723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.575855 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.575742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.576126 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.576110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.576352 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.576330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.576443 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.576417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.578066 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.578046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.583852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.583827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9dd\" (UniqueName: \"kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.700388 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.700358    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:43.831941 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:43.831847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:48:43.834600 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:48:43.834574    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdf8fcb_2e81_4106_a098_eb582b1d24dc.slice/crio-55f95c5f4f92c8b148d9c240b7cb194b484125ea30f7ef2b68f5c771795602c0 WatchSource:0}: Error finding container 55f95c5f4f92c8b148d9c240b7cb194b484125ea30f7ef2b68f5c771795602c0: Status 404 returned error can't find the container with id 55f95c5f4f92c8b148d9c240b7cb194b484125ea30f7ef2b68f5c771795602c0
Jul 28 17:48:44.041564 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:44.041467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerStarted","Data":"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"}
Jul 28 17:48:44.041564 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:44.041544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerStarted","Data":"55f95c5f4f92c8b148d9c240b7cb194b484125ea30f7ef2b68f5c771795602c0"}
Jul 28 17:48:44.043608 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:44.043585    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerID="f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f" exitCode=2
Jul 28 17:48:44.043729 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:44.043631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerDied","Data":"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"}
Jul 28 17:48:44.731520 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:44.731473    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 17:48:47.880256 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.880234    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:48:47.913999 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.913977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79zw6\" (UniqueName: \"kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6\") pod \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") "
Jul 28 17:48:47.914124 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls\") pod \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") "
Jul 28 17:48:47.914124 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location\") pod \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") "
Jul 28 17:48:47.914124 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert\") pod \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") "
Jul 28 17:48:47.914247 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config\") pod \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\" (UID: \"6c67e4eb-345b-48a8-ac2f-89c5c2afce38\") "
Jul 28 17:48:47.914463 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c67e4eb-345b-48a8-ac2f-89c5c2afce38" (UID: "6c67e4eb-345b-48a8-ac2f-89c5c2afce38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:47.914661 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914489    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c67e4eb-345b-48a8-ac2f-89c5c2afce38" (UID: "6c67e4eb-345b-48a8-ac2f-89c5c2afce38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:47.914742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.914655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c67e4eb-345b-48a8-ac2f-89c5c2afce38" (UID: "6c67e4eb-345b-48a8-ac2f-89c5c2afce38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:47.916175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.916153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6" (OuterVolumeSpecName: "kube-api-access-79zw6") pod "6c67e4eb-345b-48a8-ac2f-89c5c2afce38" (UID: "6c67e4eb-345b-48a8-ac2f-89c5c2afce38"). InnerVolumeSpecName "kube-api-access-79zw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:47.916175 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:47.916165    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c67e4eb-345b-48a8-ac2f-89c5c2afce38" (UID: "6c67e4eb-345b-48a8-ac2f-89c5c2afce38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:48.015656 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.015631    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:48:48.015656 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.015652    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zw6\" (UniqueName: \"kubernetes.io/projected/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kube-api-access-79zw6\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:48:48.015656 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.015661    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:48:48.015866 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.015670    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:48:48.015866 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.015678    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e4eb-345b-48a8-ac2f-89c5c2afce38-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:48:48.061835 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.061804    2561 generic.go:356] "Generic (PLEG): container finished" podID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerID="f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481" exitCode=0
Jul 28 17:48:48.061989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.061869    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"
Jul 28 17:48:48.061989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.061889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerDied","Data":"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"}
Jul 28 17:48:48.061989 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.061965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls" event={"ID":"6c67e4eb-345b-48a8-ac2f-89c5c2afce38","Type":"ContainerDied","Data":"5eb61cfd4f2b0f929589a83d2b61447a6006dcb0090fd51541cbb063fbc31c86"}
Jul 28 17:48:48.062158 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.061991    2561 scope.go:117] "RemoveContainer" containerID="f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"
Jul 28 17:48:48.063345 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.063324    2561 generic.go:356] "Generic (PLEG): container finished" podID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerID="f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061" exitCode=0
Jul 28 17:48:48.063448 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.063391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerDied","Data":"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"}
Jul 28 17:48:48.070818 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.070743    2561 scope.go:117] "RemoveContainer" containerID="f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"
Jul 28 17:48:48.078898 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.078881    2561 scope.go:117] "RemoveContainer" containerID="29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"
Jul 28 17:48:48.087854 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.087778    2561 scope.go:117] "RemoveContainer" containerID="f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"
Jul 28 17:48:48.088265 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:48:48.088158    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f\": container with ID starting with f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f not found: ID does not exist" containerID="f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"
Jul 28 17:48:48.088265 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.088187    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f"} err="failed to get container status \"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f\": rpc error: code = NotFound desc = could not find container \"f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f\": container with ID starting with f3025a4fcba08e3ef29792895a692d1362a363e39c42c5a0fd36b79479e23c1f not found: ID does not exist"
Jul 28 17:48:48.088265 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.088209    2561 scope.go:117] "RemoveContainer" containerID="f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"
Jul 28 17:48:48.088797 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:48:48.088776    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481\": container with ID starting with f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481 not found: ID does not exist" containerID="f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"
Jul 28 17:48:48.088875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.088805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481"} err="failed to get container status \"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481\": rpc error: code = NotFound desc = could not find container \"f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481\": container with ID starting with f717b71266701977642c5215886cb6214086cabdd5865e89c3794774ed536481 not found: ID does not exist"
Jul 28 17:48:48.088875 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.088828    2561 scope.go:117] "RemoveContainer" containerID="29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"
Jul 28 17:48:48.089543 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:48:48.089262    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f\": container with ID starting with 29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f not found: ID does not exist" containerID="29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"
Jul 28 17:48:48.089669 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.089291    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f"} err="failed to get container status \"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f\": rpc error: code = NotFound desc = could not find container \"29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f\": container with ID starting with 29779fe8b53bd460430ef9546b8f885974d34677c667b70df42dbe52e5ce862f not found: ID does not exist"
Jul 28 17:48:48.103093 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.103072    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:48:48.108853 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.108832    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls"]
Jul 28 17:48:48.204276 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:48.204245    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" path="/var/lib/kubelet/pods/6c67e4eb-345b-48a8-ac2f-89c5c2afce38/volumes"
Jul 28 17:48:49.073871 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:49.073835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerStarted","Data":"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"}
Jul 28 17:48:49.074322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:49.073926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerStarted","Data":"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"}
Jul 28 17:48:49.074322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:49.074117    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:49.097852 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:49.097809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podStartSLOduration=2.337419437 podStartE2EDuration="6.097794364s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760374927s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:49.096415754 +0000 UTC m=+1897.413854642" watchObservedRunningTime="2026-07-28 17:48:49.097794364 +0000 UTC m=+1897.415233250"
Jul 28 17:48:50.078905 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:50.078872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:50.080206 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:50.080175    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:48:51.083139 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:51.083097    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:48:56.087514 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:56.087483    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:48:56.088085 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:48:56.088062    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:06.088090 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:06.088050    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:16.091901 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:16.091774    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:26.088588 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:26.088548    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:36.088826 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:36.088780    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:46.088795 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:46.088757    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:49:56.088520 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:49:56.088483    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:50:06.088551 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:06.088469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:50:13.558162 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:13.558130    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:50:13.558751 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:13.558501    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" containerID="cri-o://e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed" gracePeriod=30
Jul 28 17:50:13.558751 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:13.558523    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kube-rbac-proxy" containerID="cri-o://b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10" gracePeriod=30
Jul 28 17:50:14.400156 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:14.400121    2561 generic.go:356] "Generic (PLEG): container finished" podID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerID="b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10" exitCode=2
Jul 28 17:50:14.400362 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:14.400193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerDied","Data":"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"}
Jul 28 17:50:16.083393 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:16.083355    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 28 17:50:16.088351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:16.088291    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:50:18.399636 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.399607    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:50:18.417445 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.417418    2561 generic.go:356] "Generic (PLEG): container finished" podID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerID="e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed" exitCode=0
Jul 28 17:50:18.417602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.417464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerDied","Data":"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"}
Jul 28 17:50:18.417602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.417470    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"
Jul 28 17:50:18.417602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.417524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw" event={"ID":"ecdf8fcb-2e81-4106-a098-eb582b1d24dc","Type":"ContainerDied","Data":"55f95c5f4f92c8b148d9c240b7cb194b484125ea30f7ef2b68f5c771795602c0"}
Jul 28 17:50:18.417602 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.417564    2561 scope.go:117] "RemoveContainer" containerID="b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"
Jul 28 17:50:18.427555 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.427538    2561 scope.go:117] "RemoveContainer" containerID="e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"
Jul 28 17:50:18.435047 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.435028    2561 scope.go:117] "RemoveContainer" containerID="f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"
Jul 28 17:50:18.442419 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.442398    2561 scope.go:117] "RemoveContainer" containerID="b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"
Jul 28 17:50:18.442685 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:50:18.442665    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10\": container with ID starting with b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10 not found: ID does not exist" containerID="b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"
Jul 28 17:50:18.442736 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.442695    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10"} err="failed to get container status \"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10\": rpc error: code = NotFound desc = could not find container \"b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10\": container with ID starting with b3887af691fbdc3c72eff0275ce0dd47c7840bd456ec0a7529b1e9e77b6d0b10 not found: ID does not exist"
Jul 28 17:50:18.442736 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.442716    2561 scope.go:117] "RemoveContainer" containerID="e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"
Jul 28 17:50:18.442951 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:50:18.442934    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed\": container with ID starting with e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed not found: ID does not exist" containerID="e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"
Jul 28 17:50:18.442996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.442958    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed"} err="failed to get container status \"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed\": rpc error: code = NotFound desc = could not find container \"e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed\": container with ID starting with e40ab89c9498ec6813087ac208098ae796734511690bd460eb3759d4dfdcdfed not found: ID does not exist"
Jul 28 17:50:18.442996 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.442976    2561 scope.go:117] "RemoveContainer" containerID="f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"
Jul 28 17:50:18.443189 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:50:18.443174    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061\": container with ID starting with f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061 not found: ID does not exist" containerID="f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"
Jul 28 17:50:18.443236 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.443195    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061"} err="failed to get container status \"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061\": rpc error: code = NotFound desc = could not find container \"f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061\": container with ID starting with f1fa6120f23dce68ca41a0d3b8e5c2c9ec8b4a54393708bdbb5856326b534061 not found: ID does not exist"
Jul 28 17:50:18.492271 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert\") pod \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") "
Jul 28 17:50:18.492475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config\") pod \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") "
Jul 28 17:50:18.492475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k9dd\" (UniqueName: \"kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd\") pod \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") "
Jul 28 17:50:18.492475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492414    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location\") pod \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") "
Jul 28 17:50:18.492475 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls\") pod \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\" (UID: \"ecdf8fcb-2e81-4106-a098-eb582b1d24dc\") "
Jul 28 17:50:18.492742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492716    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecdf8fcb-2e81-4106-a098-eb582b1d24dc" (UID: "ecdf8fcb-2e81-4106-a098-eb582b1d24dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:18.492742 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492729    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecdf8fcb-2e81-4106-a098-eb582b1d24dc" (UID: "ecdf8fcb-2e81-4106-a098-eb582b1d24dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:18.492823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.492772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecdf8fcb-2e81-4106-a098-eb582b1d24dc" (UID: "ecdf8fcb-2e81-4106-a098-eb582b1d24dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:18.494534 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.494510    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecdf8fcb-2e81-4106-a098-eb582b1d24dc" (UID: "ecdf8fcb-2e81-4106-a098-eb582b1d24dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:18.494641 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.494610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd" (OuterVolumeSpecName: "kube-api-access-9k9dd") pod "ecdf8fcb-2e81-4106-a098-eb582b1d24dc" (UID: "ecdf8fcb-2e81-4106-a098-eb582b1d24dc"). InnerVolumeSpecName "kube-api-access-9k9dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:18.593863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.593825    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.593863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.593854    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9dd\" (UniqueName: \"kubernetes.io/projected/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kube-api-access-9k9dd\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.593863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.593864    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.593863 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.593873    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.594132 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.593883    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf8fcb-2e81-4106-a098-eb582b1d24dc-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.707329 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707226    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:50:18.707574 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707561    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kube-rbac-proxy"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707575    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kube-rbac-proxy"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707587    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kube-rbac-proxy"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707593    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kube-rbac-proxy"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707599    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="storage-initializer"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707606    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="storage-initializer"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707613    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container"
Jul 28 17:50:18.707621 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707618    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="storage-initializer"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707635    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="storage-initializer"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707642    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707647    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707695    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kserve-container"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707704    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kube-rbac-proxy"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c67e4eb-345b-48a8-ac2f-89c5c2afce38" containerName="kserve-container"
Jul 28 17:50:18.707908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.707716    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" containerName="kube-rbac-proxy"
Jul 28 17:50:18.712198 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.712177    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:50:18.712324 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.712295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.714931 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.714911    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:50:18.715054 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.714914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:50:18.715389 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.715372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.717593 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.717575    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 17:50:18.722374 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.722350    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:50:18.725767 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.725744    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:50:18.767557 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.767522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:50:18.773267 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.773237    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw"]
Jul 28 17:50:18.795399 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7tv\" (UniqueName: \"kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.795522 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.795572 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.795572 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznqq\" (UniqueName: \"kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.795671 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.795671 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.795741 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.795741 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.795823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.795823 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.795768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.896542 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.896542 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7tv\" (UniqueName: \"kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznqq\" (UniqueName: \"kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.896805 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.897174 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.896828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.897234 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.897432 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.897511 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.897511 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.897511 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.897796 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.897777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.899610 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.899592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:18.899685 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.899615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.904624 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.904592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7tv\" (UniqueName: \"kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-vh2l8\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:18.904721 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:18.904652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznqq\" (UniqueName: \"kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-h2x8f\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:19.026034 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.025949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:19.034242 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.034217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:19.170514 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.170484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:50:19.172497 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:50:19.172469    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b0d8d_9e59_43da_9040_22179a7026a9.slice/crio-5538ea7d9d6c284d5d63e1e628ef194447802fb0119be5df91dd296b60893a46 WatchSource:0}: Error finding container 5538ea7d9d6c284d5d63e1e628ef194447802fb0119be5df91dd296b60893a46: Status 404 returned error can't find the container with id 5538ea7d9d6c284d5d63e1e628ef194447802fb0119be5df91dd296b60893a46
Jul 28 17:50:19.188882 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.188854    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:50:19.191746 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:50:19.191723    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f22211_85c9_44b3_891b_fbdc303fe8ba.slice/crio-33f47349b4a156436d12f0ef6e74bb0eac6c1424f3cff4ae8bdddf6390399139 WatchSource:0}: Error finding container 33f47349b4a156436d12f0ef6e74bb0eac6c1424f3cff4ae8bdddf6390399139: Status 404 returned error can't find the container with id 33f47349b4a156436d12f0ef6e74bb0eac6c1424f3cff4ae8bdddf6390399139
Jul 28 17:50:19.426498 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.426457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerStarted","Data":"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"}
Jul 28 17:50:19.426986 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.426541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerStarted","Data":"33f47349b4a156436d12f0ef6e74bb0eac6c1424f3cff4ae8bdddf6390399139"}
Jul 28 17:50:19.428088 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.428057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerStarted","Data":"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"}
Jul 28 17:50:19.428195 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:19.428123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerStarted","Data":"5538ea7d9d6c284d5d63e1e628ef194447802fb0119be5df91dd296b60893a46"}
Jul 28 17:50:20.204560 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:20.204528    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecdf8fcb-2e81-4106-a098-eb582b1d24dc" path="/var/lib/kubelet/pods/ecdf8fcb-2e81-4106-a098-eb582b1d24dc/volumes"
Jul 28 17:50:23.445847 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:23.445812    2561 generic.go:356] "Generic (PLEG): container finished" podID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerID="5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac" exitCode=0
Jul 28 17:50:23.446418 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:23.445882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerDied","Data":"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"}
Jul 28 17:50:23.447561 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:23.447542    2561 generic.go:356] "Generic (PLEG): container finished" podID="784b0d8d-9e59-43da-9040-22179a7026a9" containerID="a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b" exitCode=0
Jul 28 17:50:23.447695 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:23.447575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerDied","Data":"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"}
Jul 28 17:50:24.454113 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.454071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerStarted","Data":"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"}
Jul 28 17:50:24.454594 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.454147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerStarted","Data":"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"}
Jul 28 17:50:24.454594 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.454431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:24.454594 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.454564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:24.455930 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.455901    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:50:24.456369 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.456349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerStarted","Data":"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"}
Jul 28 17:50:24.456449 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.456397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerStarted","Data":"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"}
Jul 28 17:50:24.456649 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.456633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:24.456708 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.456687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:24.457707 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.457684    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:50:24.477368 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.477323    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podStartSLOduration=2.73282246 podStartE2EDuration="6.477287998s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.744465538s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:24.475756465 +0000 UTC m=+1992.793195364" watchObservedRunningTime="2026-07-28 17:50:24.477287998 +0000 UTC m=+1992.794726886"
Jul 28 17:50:24.501666 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:24.501601    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podStartSLOduration=2.874258343 podStartE2EDuration="6.501584311s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.627325968s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:24.500632975 +0000 UTC m=+1992.818071863" watchObservedRunningTime="2026-07-28 17:50:24.501584311 +0000 UTC m=+1992.819023200"
Jul 28 17:50:25.460552 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:25.460515    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:50:25.461041 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:25.460654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:50:30.465222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:30.465169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:50:30.466143 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:30.466124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:50:30.466247 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:30.466118    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:50:30.466473 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:30.466443    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:50:40.466113 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:40.466075    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:50:40.466589 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:40.466564    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:50:50.467090 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:50.467052    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:50:50.467581 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:50:50.467051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:00.466974 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:00.466923    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:51:00.467515 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:00.466925    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:10.466484 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:10.466434    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:10.466484 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:10.466466    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:51:20.466159 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:20.466123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:51:20.466581 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:20.466423    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:30.467081 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:30.467042    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:30.467625 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:30.467045    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:51:40.466518 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:40.466423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:51:40.467010 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:40.466992    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:51:48.750810 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.750778    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:51:48.751273 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.751148    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" containerID="cri-o://46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3" gracePeriod=30
Jul 28 17:51:48.751438 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.751389    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kube-rbac-proxy" containerID="cri-o://511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983" gracePeriod=30
Jul 28 17:51:48.787460 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.787430    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:51:48.787744 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.787708    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kube-rbac-proxy" containerID="cri-o://a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9" gracePeriod=30
Jul 28 17:51:48.787898 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:48.787699    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" containerID="cri-o://8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db" gracePeriod=30
Jul 28 17:51:49.816263 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:49.816233    2561 generic.go:356] "Generic (PLEG): container finished" podID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerID="a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9" exitCode=2
Jul 28 17:51:49.816694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:49.816334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerDied","Data":"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"}
Jul 28 17:51:49.818351 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:49.818331    2561 generic.go:356] "Generic (PLEG): container finished" podID="784b0d8d-9e59-43da-9040-22179a7026a9" containerID="511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983" exitCode=2
Jul 28 17:51:49.818446 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:49.818390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerDied","Data":"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"}
Jul 28 17:51:50.461812 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:50.461766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 28 17:51:50.461983 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:50.461773    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 28 17:51:50.466694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:50.466675    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 17:51:50.466694 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:50.466686    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:51:53.543593 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.543567    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:51:53.590176 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.590150    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:51:53.596112 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location\") pod \"34f22211-85c9-44b3-891b-fbdc303fe8ba\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") "
Jul 28 17:51:53.596233 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config\") pod \"34f22211-85c9-44b3-891b-fbdc303fe8ba\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") "
Jul 28 17:51:53.596233 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596147    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls\") pod \"34f22211-85c9-44b3-891b-fbdc303fe8ba\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") "
Jul 28 17:51:53.596233 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznqq\" (UniqueName: \"kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq\") pod \"34f22211-85c9-44b3-891b-fbdc303fe8ba\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") "
Jul 28 17:51:53.596233 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert\") pod \"34f22211-85c9-44b3-891b-fbdc303fe8ba\" (UID: \"34f22211-85c9-44b3-891b-fbdc303fe8ba\") "
Jul 28 17:51:53.596500 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596473    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34f22211-85c9-44b3-891b-fbdc303fe8ba" (UID: "34f22211-85c9-44b3-891b-fbdc303fe8ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:53.596573 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596526    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34f22211-85c9-44b3-891b-fbdc303fe8ba" (UID: "34f22211-85c9-44b3-891b-fbdc303fe8ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:53.596613 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.596591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34f22211-85c9-44b3-891b-fbdc303fe8ba" (UID: "34f22211-85c9-44b3-891b-fbdc303fe8ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:53.598159 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.598138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34f22211-85c9-44b3-891b-fbdc303fe8ba" (UID: "34f22211-85c9-44b3-891b-fbdc303fe8ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:53.598206 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.598177    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq" (OuterVolumeSpecName: "kube-api-access-hznqq") pod "34f22211-85c9-44b3-891b-fbdc303fe8ba" (UID: "34f22211-85c9-44b3-891b-fbdc303fe8ba"). InnerVolumeSpecName "kube-api-access-hznqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:53.697114 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config\") pod \"784b0d8d-9e59-43da-9040-22179a7026a9\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") "
Jul 28 17:51:53.697289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7tv\" (UniqueName: \"kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv\") pod \"784b0d8d-9e59-43da-9040-22179a7026a9\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") "
Jul 28 17:51:53.697289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697162    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location\") pod \"784b0d8d-9e59-43da-9040-22179a7026a9\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") "
Jul 28 17:51:53.697289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert\") pod \"784b0d8d-9e59-43da-9040-22179a7026a9\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") "
Jul 28 17:51:53.697289 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls\") pod \"784b0d8d-9e59-43da-9040-22179a7026a9\" (UID: \"784b0d8d-9e59-43da-9040-22179a7026a9\") "
Jul 28 17:51:53.697547 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697497    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f22211-85c9-44b3-891b-fbdc303fe8ba-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.697547 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697517    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.697547 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "784b0d8d-9e59-43da-9040-22179a7026a9" (UID: "784b0d8d-9e59-43da-9040-22179a7026a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:53.697673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697532    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34f22211-85c9-44b3-891b-fbdc303fe8ba-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.697673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "784b0d8d-9e59-43da-9040-22179a7026a9" (UID: "784b0d8d-9e59-43da-9040-22179a7026a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:53.697673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697581    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznqq\" (UniqueName: \"kubernetes.io/projected/34f22211-85c9-44b3-891b-fbdc303fe8ba-kube-api-access-hznqq\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.697673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697592    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34f22211-85c9-44b3-891b-fbdc303fe8ba-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.697673 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.697597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "784b0d8d-9e59-43da-9040-22179a7026a9" (UID: "784b0d8d-9e59-43da-9040-22179a7026a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:53.699319 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.699286    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "784b0d8d-9e59-43da-9040-22179a7026a9" (UID: "784b0d8d-9e59-43da-9040-22179a7026a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:53.699379 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.699331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv" (OuterVolumeSpecName: "kube-api-access-tq7tv") pod "784b0d8d-9e59-43da-9040-22179a7026a9" (UID: "784b0d8d-9e59-43da-9040-22179a7026a9"). InnerVolumeSpecName "kube-api-access-tq7tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:53.798890 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.798855    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.798890 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.798892    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7tv\" (UniqueName: \"kubernetes.io/projected/784b0d8d-9e59-43da-9040-22179a7026a9-kube-api-access-tq7tv\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.799083 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.798905    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784b0d8d-9e59-43da-9040-22179a7026a9-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.799083 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.798916    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784b0d8d-9e59-43da-9040-22179a7026a9-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.799083 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.798925    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784b0d8d-9e59-43da-9040-22179a7026a9-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:51:53.839058 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.839019    2561 generic.go:356] "Generic (PLEG): container finished" podID="784b0d8d-9e59-43da-9040-22179a7026a9" containerID="46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3" exitCode=0
Jul 28 17:51:53.839213 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.839094    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"
Jul 28 17:51:53.839213 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.839101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerDied","Data":"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"}
Jul 28 17:51:53.839213 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.839161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8" event={"ID":"784b0d8d-9e59-43da-9040-22179a7026a9","Type":"ContainerDied","Data":"5538ea7d9d6c284d5d63e1e628ef194447802fb0119be5df91dd296b60893a46"}
Jul 28 17:51:53.839213 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.839190    2561 scope.go:117] "RemoveContainer" containerID="511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"
Jul 28 17:51:53.841222 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.841194    2561 generic.go:356] "Generic (PLEG): container finished" podID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerID="8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db" exitCode=0
Jul 28 17:51:53.841362 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.841250    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"
Jul 28 17:51:53.841362 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.841247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerDied","Data":"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"}
Jul 28 17:51:53.841455 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.841370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f" event={"ID":"34f22211-85c9-44b3-891b-fbdc303fe8ba","Type":"ContainerDied","Data":"33f47349b4a156436d12f0ef6e74bb0eac6c1424f3cff4ae8bdddf6390399139"}
Jul 28 17:51:53.848646 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.848629    2561 scope.go:117] "RemoveContainer" containerID="46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"
Jul 28 17:51:53.856198 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.856181    2561 scope.go:117] "RemoveContainer" containerID="a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"
Jul 28 17:51:53.862804 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.862784    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:51:53.863492 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.863478    2561 scope.go:117] "RemoveContainer" containerID="511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"
Jul 28 17:51:53.863751 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.863731    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983\": container with ID starting with 511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983 not found: ID does not exist" containerID="511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"
Jul 28 17:51:53.863812 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.863758    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983"} err="failed to get container status \"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983\": rpc error: code = NotFound desc = could not find container \"511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983\": container with ID starting with 511fc5b0f5a9a38a332ef10662ac8d7331f31a819823098027040ad72b6de983 not found: ID does not exist"
Jul 28 17:51:53.863812 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.863773    2561 scope.go:117] "RemoveContainer" containerID="46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"
Jul 28 17:51:53.863984 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.863970    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3\": container with ID starting with 46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3 not found: ID does not exist" containerID="46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"
Jul 28 17:51:53.864026 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.863987    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3"} err="failed to get container status \"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3\": rpc error: code = NotFound desc = could not find container \"46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3\": container with ID starting with 46216c958074a2c732177e77f366a21fd028db746dbca57fff00d4d5bcc0c0b3 not found: ID does not exist"
Jul 28 17:51:53.864026 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.864000    2561 scope.go:117] "RemoveContainer" containerID="a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"
Jul 28 17:51:53.864243 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.864226    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b\": container with ID starting with a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b not found: ID does not exist" containerID="a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"
Jul 28 17:51:53.864322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.864250    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b"} err="failed to get container status \"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b\": rpc error: code = NotFound desc = could not find container \"a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b\": container with ID starting with a2996bb06e399f7c5d39f2d1146f1c5ed2d8d06d552627f63fb594f4d087a15b not found: ID does not exist"
Jul 28 17:51:53.864322 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.864270    2561 scope.go:117] "RemoveContainer" containerID="a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"
Jul 28 17:51:53.868516 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.868497    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8"]
Jul 28 17:51:53.871985 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.871963    2561 scope.go:117] "RemoveContainer" containerID="8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"
Jul 28 17:51:53.879391 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.879372    2561 scope.go:117] "RemoveContainer" containerID="5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"
Jul 28 17:51:53.882414 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.882391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:51:53.886938 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.886917    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f"]
Jul 28 17:51:53.888163 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.888142    2561 scope.go:117] "RemoveContainer" containerID="a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"
Jul 28 17:51:53.888441 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.888421    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9\": container with ID starting with a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9 not found: ID does not exist" containerID="a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"
Jul 28 17:51:53.888497 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.888451    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9"} err="failed to get container status \"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9\": rpc error: code = NotFound desc = could not find container \"a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9\": container with ID starting with a5549292eaefbc52d8947aa220559a8591822bca10fec7375024d28332ccecc9 not found: ID does not exist"
Jul 28 17:51:53.888497 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.888469    2561 scope.go:117] "RemoveContainer" containerID="8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"
Jul 28 17:51:53.888735 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.888714    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db\": container with ID starting with 8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db not found: ID does not exist" containerID="8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"
Jul 28 17:51:53.888789 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.888740    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db"} err="failed to get container status \"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db\": rpc error: code = NotFound desc = could not find container \"8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db\": container with ID starting with 8d30c4827d369ddc3cffa7dbabfb0cc22742e480ba607c569c4b37f6520d42db not found: ID does not exist"
Jul 28 17:51:53.888789 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.888756    2561 scope.go:117] "RemoveContainer" containerID="5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"
Jul 28 17:51:53.889006 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:51:53.888991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac\": container with ID starting with 5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac not found: ID does not exist" containerID="5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"
Jul 28 17:51:53.889055 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:53.889009    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac"} err="failed to get container status \"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac\": rpc error: code = NotFound desc = could not find container \"5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac\": container with ID starting with 5762d4432d8a6a6d448f4810dfc913d841d4a62041db3e1c7ab4ca945d94b7ac not found: ID does not exist"
Jul 28 17:51:54.204112 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:54.204082    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" path="/var/lib/kubelet/pods/34f22211-85c9-44b3-891b-fbdc303fe8ba/volumes"
Jul 28 17:51:54.204595 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:51:54.204579    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" path="/var/lib/kubelet/pods/784b0d8d-9e59-43da-9040-22179a7026a9/volumes"
Jul 28 17:53:34.387249 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387217    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387529    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387540    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387553    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="storage-initializer"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387559    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="storage-initializer"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387569    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387576    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387582    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387587    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="storage-initializer"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="storage-initializer"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387609    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387614    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387665    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kserve-container"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387673    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387680    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="784b0d8d-9e59-43da-9040-22179a7026a9" containerName="kube-rbac-proxy"
Jul 28 17:53:34.387739 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.387685    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f22211-85c9-44b3-891b-fbdc303fe8ba" containerName="kserve-container"
Jul 28 17:53:34.390640 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.390624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.393413 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.393394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:53:34.393580 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.393567    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 17:53:34.393633 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.393613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:53:34.394496 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.394465    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:53:34.394590 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.394469    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:53:34.394590 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.394517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:53:34.394590 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.394561    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:53:34.402626 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.402600    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:53:34.504820 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.504779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp9f\" (UniqueName: \"kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.504820 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.504819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.505091 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.504856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.505091 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.504980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.505091 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.505057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.606450 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.606417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp9f\" (UniqueName: \"kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.606450 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.606452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.606728 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.606477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.606728 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.606517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.606728 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.606556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.607028 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.607000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.607231 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.607209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.607421 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.607405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.609097 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.609077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.616469 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.616447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp9f\" (UniqueName: \"kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.700566 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.700533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:34.833690 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:34.833484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:53:34.835828 ip-10-0-138-92 kubenswrapper[2561]: W0728 17:53:34.835797    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d06fb1_b87e_4500_882d_c70bb49f6be0.slice/crio-3c94b021651fab089673b761a58e969188aee4e697eba676a9762eaed014c5a3 WatchSource:0}: Error finding container 3c94b021651fab089673b761a58e969188aee4e697eba676a9762eaed014c5a3: Status 404 returned error can't find the container with id 3c94b021651fab089673b761a58e969188aee4e697eba676a9762eaed014c5a3
Jul 28 17:53:35.236107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:35.236071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerStarted","Data":"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"}
Jul 28 17:53:35.236335 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:35.236127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerStarted","Data":"3c94b021651fab089673b761a58e969188aee4e697eba676a9762eaed014c5a3"}
Jul 28 17:53:39.252699 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:39.252663    2561 generic.go:356] "Generic (PLEG): container finished" podID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerID="59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99" exitCode=0
Jul 28 17:53:39.253142 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:39.252733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerDied","Data":"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"}
Jul 28 17:53:39.253934 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:39.253921    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:53:46.289047 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.289007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerStarted","Data":"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"}
Jul 28 17:53:46.289603 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.289087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerStarted","Data":"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"}
Jul 28 17:53:46.289603 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.289385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:46.289603 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.289443    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:46.290717 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.290693    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:53:46.309426 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:46.309364    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podStartSLOduration=2.421361036 podStartE2EDuration="12.309347581s" totalImagesPullingTime="6.293433556s" totalInitContainerRuntime="3.594552989s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:46.30767278 +0000 UTC m=+2194.625111668" watchObservedRunningTime="2026-07-28 17:53:46.309347581 +0000 UTC m=+2194.626786469"
Jul 28 17:53:47.293050 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:47.293006    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:53:52.298084 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:52.298055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:53:52.298571 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:53:52.298538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:02.299432 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:02.299395    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:12.299347 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:12.299298    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:22.299537 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:22.299498    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:32.299164 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:32.299127    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:42.298785 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:42.298703    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:54:52.299390 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:54:52.299343    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 17:55:02.299459 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:02.299430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:55:04.913785 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:04.913751    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:55:04.914384 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:04.914018    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container" containerID="cri-o://ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037" gracePeriod=30
Jul 28 17:55:04.914384 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:04.914091    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kube-rbac-proxy" containerID="cri-o://fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb" gracePeriod=30
Jul 28 17:55:05.611404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:05.611373    2561 generic.go:356] "Generic (PLEG): container finished" podID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerID="fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb" exitCode=2
Jul 28 17:55:05.611579 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:05.611423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerDied","Data":"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"}
Jul 28 17:55:07.293424 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:07.293383    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 28 17:55:09.268734 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.268708    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:55:09.399908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.399836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwp9f\" (UniqueName: \"kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f\") pod \"56d06fb1-b87e-4500-882d-c70bb49f6be0\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") "
Jul 28 17:55:09.399908 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.399867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert\") pod \"56d06fb1-b87e-4500-882d-c70bb49f6be0\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") "
Jul 28 17:55:09.400107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.399921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config\") pod \"56d06fb1-b87e-4500-882d-c70bb49f6be0\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") "
Jul 28 17:55:09.400107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.399949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls\") pod \"56d06fb1-b87e-4500-882d-c70bb49f6be0\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") "
Jul 28 17:55:09.400107 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.399975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location\") pod \"56d06fb1-b87e-4500-882d-c70bb49f6be0\" (UID: \"56d06fb1-b87e-4500-882d-c70bb49f6be0\") "
Jul 28 17:55:09.400299 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.400273    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56d06fb1-b87e-4500-882d-c70bb49f6be0" (UID: "56d06fb1-b87e-4500-882d-c70bb49f6be0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:09.400404 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.400355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56d06fb1-b87e-4500-882d-c70bb49f6be0" (UID: "56d06fb1-b87e-4500-882d-c70bb49f6be0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:09.400462 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.400420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56d06fb1-b87e-4500-882d-c70bb49f6be0" (UID: "56d06fb1-b87e-4500-882d-c70bb49f6be0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:09.401876 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.401855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f" (OuterVolumeSpecName: "kube-api-access-zwp9f") pod "56d06fb1-b87e-4500-882d-c70bb49f6be0" (UID: "56d06fb1-b87e-4500-882d-c70bb49f6be0"). InnerVolumeSpecName "kube-api-access-zwp9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:09.401930 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.401893    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56d06fb1-b87e-4500-882d-c70bb49f6be0" (UID: "56d06fb1-b87e-4500-882d-c70bb49f6be0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:09.500731 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.500701    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:55:09.500731 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.500730    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d06fb1-b87e-4500-882d-c70bb49f6be0-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:55:09.500923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.500741    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56d06fb1-b87e-4500-882d-c70bb49f6be0-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:55:09.500923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.500750    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwp9f\" (UniqueName: \"kubernetes.io/projected/56d06fb1-b87e-4500-882d-c70bb49f6be0-kube-api-access-zwp9f\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:55:09.500923 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.500758    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56d06fb1-b87e-4500-882d-c70bb49f6be0-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 17:55:09.627653 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.627619    2561 generic.go:356] "Generic (PLEG): container finished" podID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerID="ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037" exitCode=0
Jul 28 17:55:09.627793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.627679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"
Jul 28 17:55:09.627793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.627703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerDied","Data":"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"}
Jul 28 17:55:09.627793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.627762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r" event={"ID":"56d06fb1-b87e-4500-882d-c70bb49f6be0","Type":"ContainerDied","Data":"3c94b021651fab089673b761a58e969188aee4e697eba676a9762eaed014c5a3"}
Jul 28 17:55:09.627793 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.627789    2561 scope.go:117] "RemoveContainer" containerID="fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"
Jul 28 17:55:09.636119 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.636103    2561 scope.go:117] "RemoveContainer" containerID="ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"
Jul 28 17:55:09.643201 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.643185    2561 scope.go:117] "RemoveContainer" containerID="59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"
Jul 28 17:55:09.650161 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650115    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:55:09.650401 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650386    2561 scope.go:117] "RemoveContainer" containerID="fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"
Jul 28 17:55:09.650638 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:55:09.650619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb\": container with ID starting with fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb not found: ID does not exist" containerID="fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"
Jul 28 17:55:09.650676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb"} err="failed to get container status \"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb\": rpc error: code = NotFound desc = could not find container \"fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb\": container with ID starting with fcc472dceb22ddbbad281c801e6f64c6564b1ae7c75528cf105becbbf7dec6eb not found: ID does not exist"
Jul 28 17:55:09.650676 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650663    2561 scope.go:117] "RemoveContainer" containerID="ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"
Jul 28 17:55:09.650868 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:55:09.650853    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037\": container with ID starting with ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037 not found: ID does not exist" containerID="ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"
Jul 28 17:55:09.650904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650873    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037"} err="failed to get container status \"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037\": rpc error: code = NotFound desc = could not find container \"ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037\": container with ID starting with ae9d96f4e290ddfa11fb2aad52cc132e5f4524993708dbfefc636114cbaba037 not found: ID does not exist"
Jul 28 17:55:09.650904 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.650886    2561 scope.go:117] "RemoveContainer" containerID="59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"
Jul 28 17:55:09.651102 ip-10-0-138-92 kubenswrapper[2561]: E0728 17:55:09.651088    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99\": container with ID starting with 59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99 not found: ID does not exist" containerID="59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"
Jul 28 17:55:09.651135 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.651106    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99"} err="failed to get container status \"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99\": rpc error: code = NotFound desc = could not find container \"59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99\": container with ID starting with 59fab28aa7354c765d7eba4e75c155353657156a6e9b72762464f30e0906ca99 not found: ID does not exist"
Jul 28 17:55:09.654017 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:09.653998    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r"]
Jul 28 17:55:10.203240 ip-10-0-138-92 kubenswrapper[2561]: I0728 17:55:10.203201    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" path="/var/lib/kubelet/pods/56d06fb1-b87e-4500-882d-c70bb49f6be0/volumes"
Jul 28 18:01:01.191257 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191173    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191481    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="storage-initializer"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191492    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="storage-initializer"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191503    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191509    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191527    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kube-rbac-proxy"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191533    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kube-rbac-proxy"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kserve-container"
Jul 28 18:01:01.191764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.191590    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d06fb1-b87e-4500-882d-c70bb49f6be0" containerName="kube-rbac-proxy"
Jul 28 18:01:01.194806 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.194790    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.197486 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.197467    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:01:01.197946 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.197927    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 18:01:01.198000 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.197933    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:01:01.198248 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.198233    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:01:01.198555 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.198521    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 18:01:01.198621 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.198561    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:01:01.198714 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.198524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:01:01.205968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.205948    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:01:01.221450 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.221425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmcs\" (UniqueName: \"kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.221566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.221465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.221626 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.221607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.221665 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.221637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.221704 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.221679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.322561 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.322526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.322765 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.322618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.322765 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.322647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.322765 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.322688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.322765 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.322715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmcs\" (UniqueName: \"kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.323140 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.323114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.323371 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.323355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.323434 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.323417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.324938 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.324921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.345273 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.345253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmcs\" (UniqueName: \"kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-shpq6\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.504776 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.504686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:01.646097 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.646071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:01:01.648506 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:01:01.648477    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3f589f_45f3_4634_a20b_eaf2296f0877.slice/crio-186e0b4f8299526ec750a389698f04bc0e981e02f169a3f0a37becf42075081c WatchSource:0}: Error finding container 186e0b4f8299526ec750a389698f04bc0e981e02f169a3f0a37becf42075081c: Status 404 returned error can't find the container with id 186e0b4f8299526ec750a389698f04bc0e981e02f169a3f0a37becf42075081c
Jul 28 18:01:01.984171 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.984130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerStarted","Data":"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"}
Jul 28 18:01:01.984386 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:01.984193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerStarted","Data":"186e0b4f8299526ec750a389698f04bc0e981e02f169a3f0a37becf42075081c"}
Jul 28 18:01:08.009282 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:08.009244    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerID="d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb" exitCode=0
Jul 28 18:01:08.009804 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:08.009292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerDied","Data":"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"}
Jul 28 18:01:08.010524 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:08.010508    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:01:20.062960 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:20.062924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerStarted","Data":"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"}
Jul 28 18:01:20.063432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:20.063000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerStarted","Data":"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"}
Jul 28 18:01:20.063432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:20.063219    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:20.089014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:20.088949    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podStartSLOduration=1.4801846269999999 podStartE2EDuration="19.088929963s" totalImagesPullingTime="11.551081853s" totalInitContainerRuntime="6.057663483s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:20.086925078 +0000 UTC m=+2648.404363959" watchObservedRunningTime="2026-07-28 18:01:20.088929963 +0000 UTC m=+2648.406368852"
Jul 28 18:01:21.067701 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:21.067666    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:21.069041 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:21.069005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:01:22.071734 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:22.071695    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:01:27.076613 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:27.076581    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:01:27.077106 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:27.077079    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:01:37.077616 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:37.077568    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:01:47.082986 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:47.082931    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:01:57.077363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:01:57.077324    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:02:07.077460 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:07.077431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:02:12.885523 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:12.885488    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:02:12.886112 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:12.885844    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container" containerID="cri-o://e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e" gracePeriod=30
Jul 28 18:02:12.886112 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:12.885893    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kube-rbac-proxy" containerID="cri-o://b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d" gracePeriod=30
Jul 28 18:02:13.270050 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:13.270015    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerID="b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d" exitCode=2
Jul 28 18:02:13.270225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:13.270075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerDied","Data":"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"}
Jul 28 18:02:15.729793 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.729768    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:02:15.865395 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865253    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location\") pod \"8a3f589f-45f3-4634-a20b-eaf2296f0877\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") "
Jul 28 18:02:15.865395 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert\") pod \"8a3f589f-45f3-4634-a20b-eaf2296f0877\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") "
Jul 28 18:02:15.865395 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config\") pod \"8a3f589f-45f3-4634-a20b-eaf2296f0877\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") "
Jul 28 18:02:15.865703 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmmcs\" (UniqueName: \"kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs\") pod \"8a3f589f-45f3-4634-a20b-eaf2296f0877\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") "
Jul 28 18:02:15.865703 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865425    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls\") pod \"8a3f589f-45f3-4634-a20b-eaf2296f0877\" (UID: \"8a3f589f-45f3-4634-a20b-eaf2296f0877\") "
Jul 28 18:02:15.865788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865692    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a3f589f-45f3-4634-a20b-eaf2296f0877" (UID: "8a3f589f-45f3-4634-a20b-eaf2296f0877"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:02:15.865788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.865750    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a3f589f-45f3-4634-a20b-eaf2296f0877" (UID: "8a3f589f-45f3-4634-a20b-eaf2296f0877"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:02:15.867605 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.867577    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a3f589f-45f3-4634-a20b-eaf2296f0877" (UID: "8a3f589f-45f3-4634-a20b-eaf2296f0877"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:15.867713 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.867602    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs" (OuterVolumeSpecName: "kube-api-access-xmmcs") pod "8a3f589f-45f3-4634-a20b-eaf2296f0877" (UID: "8a3f589f-45f3-4634-a20b-eaf2296f0877"). InnerVolumeSpecName "kube-api-access-xmmcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:15.874747 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.874723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a3f589f-45f3-4634-a20b-eaf2296f0877" (UID: "8a3f589f-45f3-4634-a20b-eaf2296f0877"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:15.966363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.966326    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3f589f-45f3-4634-a20b-eaf2296f0877-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:02:15.966363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.966358    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:02:15.966363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.966368    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:02:15.966598 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.966376    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmcs\" (UniqueName: \"kubernetes.io/projected/8a3f589f-45f3-4634-a20b-eaf2296f0877-kube-api-access-xmmcs\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:02:15.966598 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:15.966388    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3f589f-45f3-4634-a20b-eaf2296f0877-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:02:16.284188 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.284153    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerID="e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e" exitCode=0
Jul 28 18:02:16.284384 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.284218    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"
Jul 28 18:02:16.284384 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.284244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerDied","Data":"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"}
Jul 28 18:02:16.284384 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.284331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6" event={"ID":"8a3f589f-45f3-4634-a20b-eaf2296f0877","Type":"ContainerDied","Data":"186e0b4f8299526ec750a389698f04bc0e981e02f169a3f0a37becf42075081c"}
Jul 28 18:02:16.284384 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.284363    2561 scope.go:117] "RemoveContainer" containerID="b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"
Jul 28 18:02:16.292982 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.292961    2561 scope.go:117] "RemoveContainer" containerID="e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"
Jul 28 18:02:16.301006 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.300626    2561 scope.go:117] "RemoveContainer" containerID="d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"
Jul 28 18:02:16.302637 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.302609    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:02:16.307053 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.307028    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6"]
Jul 28 18:02:16.309477 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.309457    2561 scope.go:117] "RemoveContainer" containerID="b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"
Jul 28 18:02:16.309800 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:02:16.309781    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d\": container with ID starting with b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d not found: ID does not exist" containerID="b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"
Jul 28 18:02:16.309847 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.309810    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d"} err="failed to get container status \"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d\": rpc error: code = NotFound desc = could not find container \"b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d\": container with ID starting with b28c4537db416c9e6f5d46ca2adcca41808d9ded27869dcfb7eb186848793e6d not found: ID does not exist"
Jul 28 18:02:16.309847 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.309830    2561 scope.go:117] "RemoveContainer" containerID="e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"
Jul 28 18:02:16.310099 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:02:16.310082    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e\": container with ID starting with e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e not found: ID does not exist" containerID="e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"
Jul 28 18:02:16.310172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.310109    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e"} err="failed to get container status \"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e\": rpc error: code = NotFound desc = could not find container \"e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e\": container with ID starting with e7a1448941d3c38d3577a24d9140067eeff18537f2130796050b1beeb8b4429e not found: ID does not exist"
Jul 28 18:02:16.310172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.310131    2561 scope.go:117] "RemoveContainer" containerID="d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"
Jul 28 18:02:16.310703 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:02:16.310685    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb\": container with ID starting with d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb not found: ID does not exist" containerID="d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"
Jul 28 18:02:16.310759 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:16.310708    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb"} err="failed to get container status \"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb\": rpc error: code = NotFound desc = could not find container \"d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb\": container with ID starting with d852d4f9e79cfbd11eba8b68b57614ec62f74603de3933c02bf114c2987fa8eb not found: ID does not exist"
Jul 28 18:02:18.082082 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082052    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082388    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kube-rbac-proxy"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082400    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kube-rbac-proxy"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082416    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082422    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082434    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="storage-initializer"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082439    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="storage-initializer"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082483    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kserve-container"
Jul 28 18:02:18.082495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.082491    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" containerName="kube-rbac-proxy"
Jul 28 18:02:18.085406 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.085389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.089834 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.089812    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:02:18.090162 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090146    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:02:18.090277 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090146    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:02:18.090354 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090289    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 18:02:18.090508 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090494    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:02:18.090585 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090497    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:02:18.091001 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.090983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:02:18.110830 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.110795    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:02:18.182692 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.182648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.182692 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.182688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tgx\" (UniqueName: \"kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.182925 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.182712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.182925 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.182828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.182925 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.182876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.203770 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.203736    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3f589f-45f3-4634-a20b-eaf2296f0877" path="/var/lib/kubelet/pods/8a3f589f-45f3-4634-a20b-eaf2296f0877/volumes"
Jul 28 18:02:18.283562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.283515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.283562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.283568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.283846 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:02:18.283694    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 18:02:18.283846 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.283750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tgx\" (UniqueName: \"kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.283846 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:02:18.283767    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls podName:92652119-ec90-4ba0-8b76-b3b39339427d nodeName:}" failed. No retries permitted until 2026-07-28 18:02:18.783746563 +0000 UTC m=+2707.101185444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-cvnvf" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 18:02:18.283846 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.283815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.284076 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.283924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.284288 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.284266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.284443 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.284423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.284517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.284502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.294791 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.294766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tgx\" (UniqueName: \"kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.787832 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.787779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.790439 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.790404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cvnvf\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:18.995714 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:18.995656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:19.132621 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:19.132516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:02:19.135253 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:02:19.135224    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92652119_ec90_4ba0_8b76_b3b39339427d.slice/crio-93a6d7374fd5345e20c870eae6b9605b141823e868fd5d35fd7b3e549046ab4a WatchSource:0}: Error finding container 93a6d7374fd5345e20c870eae6b9605b141823e868fd5d35fd7b3e549046ab4a: Status 404 returned error can't find the container with id 93a6d7374fd5345e20c870eae6b9605b141823e868fd5d35fd7b3e549046ab4a
Jul 28 18:02:19.300509 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:19.300405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerStarted","Data":"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"}
Jul 28 18:02:19.300509 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:19.300483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerStarted","Data":"93a6d7374fd5345e20c870eae6b9605b141823e868fd5d35fd7b3e549046ab4a"}
Jul 28 18:02:24.322219 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:24.322184    2561 generic.go:356] "Generic (PLEG): container finished" podID="92652119-ec90-4ba0-8b76-b3b39339427d" containerID="1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91" exitCode=0
Jul 28 18:02:24.322669 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:24.322242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerDied","Data":"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"}
Jul 28 18:02:25.328475 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:25.328443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerStarted","Data":"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"}
Jul 28 18:02:25.328862 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:25.328509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerStarted","Data":"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"}
Jul 28 18:02:25.328862 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:25.328697    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:25.354351 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:25.354276    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podStartSLOduration=2.6673388239999998 podStartE2EDuration="7.354261993s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.686923169s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:25.351176417 +0000 UTC m=+2713.668615305" watchObservedRunningTime="2026-07-28 18:02:25.354261993 +0000 UTC m=+2713.671700880"
Jul 28 18:02:26.333003 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:26.332970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:26.334350 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:26.334298    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:02:27.338214 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:27.338171    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:02:32.343836 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:32.343807    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:02:32.344336 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:32.344288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:02:42.345250 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:42.345206    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:02:52.344467 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:02:52.344422    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:03:02.344537 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:02.344501    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 28 18:03:12.345381 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:12.345349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:03:19.676078 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:19.676042    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:03:19.676534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:19.676326    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container" containerID="cri-o://d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d" gracePeriod=30
Jul 28 18:03:19.676534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:19.676354    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kube-rbac-proxy" containerID="cri-o://d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73" gracePeriod=30
Jul 28 18:03:20.551845 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:20.551809    2561 generic.go:356] "Generic (PLEG): container finished" podID="92652119-ec90-4ba0-8b76-b3b39339427d" containerID="d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73" exitCode=2
Jul 28 18:03:20.552016 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:20.551871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerDied","Data":"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"}
Jul 28 18:03:22.225252 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.225229    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:03:22.321974 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.321881    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config\") pod \"92652119-ec90-4ba0-8b76-b3b39339427d\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") "
Jul 28 18:03:22.321974 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.321922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert\") pod \"92652119-ec90-4ba0-8b76-b3b39339427d\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") "
Jul 28 18:03:22.321974 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.321956    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location\") pod \"92652119-ec90-4ba0-8b76-b3b39339427d\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") "
Jul 28 18:03:22.322236 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") pod \"92652119-ec90-4ba0-8b76-b3b39339427d\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") "
Jul 28 18:03:22.322236 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8tgx\" (UniqueName: \"kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx\") pod \"92652119-ec90-4ba0-8b76-b3b39339427d\" (UID: \"92652119-ec90-4ba0-8b76-b3b39339427d\") "
Jul 28 18:03:22.322405 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92652119-ec90-4ba0-8b76-b3b39339427d" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:22.322469 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92652119-ec90-4ba0-8b76-b3b39339427d" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:22.322512 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322475    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:03:22.322512 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.322495    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92652119-ec90-4ba0-8b76-b3b39339427d-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:03:22.324267 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.324241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92652119-ec90-4ba0-8b76-b3b39339427d" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:22.324267 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.324248    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx" (OuterVolumeSpecName: "kube-api-access-r8tgx") pod "92652119-ec90-4ba0-8b76-b3b39339427d" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d"). InnerVolumeSpecName "kube-api-access-r8tgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:22.332257 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.332235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92652119-ec90-4ba0-8b76-b3b39339427d" (UID: "92652119-ec90-4ba0-8b76-b3b39339427d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:22.423681 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.423649    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92652119-ec90-4ba0-8b76-b3b39339427d-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:03:22.423681 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.423677    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92652119-ec90-4ba0-8b76-b3b39339427d-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:03:22.423876 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.423692    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tgx\" (UniqueName: \"kubernetes.io/projected/92652119-ec90-4ba0-8b76-b3b39339427d-kube-api-access-r8tgx\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:03:22.562283 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.562250    2561 generic.go:356] "Generic (PLEG): container finished" podID="92652119-ec90-4ba0-8b76-b3b39339427d" containerID="d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d" exitCode=0
Jul 28 18:03:22.562429 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.562321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"
Jul 28 18:03:22.562429 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.562342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerDied","Data":"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"}
Jul 28 18:03:22.562429 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.562405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf" event={"ID":"92652119-ec90-4ba0-8b76-b3b39339427d","Type":"ContainerDied","Data":"93a6d7374fd5345e20c870eae6b9605b141823e868fd5d35fd7b3e549046ab4a"}
Jul 28 18:03:22.562554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.562432    2561 scope.go:117] "RemoveContainer" containerID="d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"
Jul 28 18:03:22.571431 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.571416    2561 scope.go:117] "RemoveContainer" containerID="d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"
Jul 28 18:03:22.578397 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.578379    2561 scope.go:117] "RemoveContainer" containerID="1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"
Jul 28 18:03:22.585472 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.585454    2561 scope.go:117] "RemoveContainer" containerID="d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"
Jul 28 18:03:22.585726 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:03:22.585705    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73\": container with ID starting with d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73 not found: ID does not exist" containerID="d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"
Jul 28 18:03:22.585809 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.585739    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73"} err="failed to get container status \"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73\": rpc error: code = NotFound desc = could not find container \"d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73\": container with ID starting with d1c1fa7d5761b370e59cafb9c9302b1879e64c98aad64e8b458187d01c6c0c73 not found: ID does not exist"
Jul 28 18:03:22.585809 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.585763    2561 scope.go:117] "RemoveContainer" containerID="d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"
Jul 28 18:03:22.585901 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.585864    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:03:22.586077 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:03:22.586061    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d\": container with ID starting with d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d not found: ID does not exist" containerID="d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"
Jul 28 18:03:22.586114 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.586084    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d"} err="failed to get container status \"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d\": rpc error: code = NotFound desc = could not find container \"d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d\": container with ID starting with d5844d7ba54ffbf66a24006737113f29a464583de785c30ceca537497c341d6d not found: ID does not exist"
Jul 28 18:03:22.586114 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.586099    2561 scope.go:117] "RemoveContainer" containerID="1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"
Jul 28 18:03:22.586373 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:03:22.586350    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91\": container with ID starting with 1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91 not found: ID does not exist" containerID="1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"
Jul 28 18:03:22.586446 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.586376    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91"} err="failed to get container status \"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91\": rpc error: code = NotFound desc = could not find container \"1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91\": container with ID starting with 1f550f68841ae908423196ba132040b03b720a80d31aa1348de782177e303f91 not found: ID does not exist"
Jul 28 18:03:22.593907 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:22.593887    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf"]
Jul 28 18:03:24.203676 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.203644    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" path="/var/lib/kubelet/pods/92652119-ec90-4ba0-8b76-b3b39339427d/volumes"
Jul 28 18:03:24.867354 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867324    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:03:24.867638 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="storage-initializer"
Jul 28 18:03:24.867688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867640    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="storage-initializer"
Jul 28 18:03:24.867688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container"
Jul 28 18:03:24.867688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867662    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container"
Jul 28 18:03:24.867688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kube-rbac-proxy"
Jul 28 18:03:24.867688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867676    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kube-rbac-proxy"
Jul 28 18:03:24.867838 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867731    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kserve-container"
Jul 28 18:03:24.867838 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.867742    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92652119-ec90-4ba0-8b76-b3b39339427d" containerName="kube-rbac-proxy"
Jul 28 18:03:24.872197 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.872177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:24.875851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.875821    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:03:24.875964 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.875894    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:03:24.876120 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.876105    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:03:24.876199 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.876143    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 18:03:24.877155 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.877134    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 18:03:24.877278 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.877170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:03:24.877278 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.877222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:03:24.884359 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.884337    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:03:24.943271 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.943243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:24.943430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.943278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8998\" (UniqueName: \"kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:24.943430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.943329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:24.943430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.943356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:24.943430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:24.943401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.043878 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.043848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044037 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.043914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044037 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.043949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8998\" (UniqueName: \"kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044037 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.043978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044037 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.044016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.044408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044692 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.044673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.044739 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.044673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.046361 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.046342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.053240 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.053216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8998\" (UniqueName: \"kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.183245 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.183218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:25.317513 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.317487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:03:25.320029 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:03:25.320004    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d7f0b9_07e3_4828_8640_ed456904ea93.slice/crio-f26f737e9e6ca6d80667db45c539fce8cc42c5375efc13ae1f183cfe61cdaa25 WatchSource:0}: Error finding container f26f737e9e6ca6d80667db45c539fce8cc42c5375efc13ae1f183cfe61cdaa25: Status 404 returned error can't find the container with id f26f737e9e6ca6d80667db45c539fce8cc42c5375efc13ae1f183cfe61cdaa25
Jul 28 18:03:25.576194 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.576106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerStarted","Data":"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"}
Jul 28 18:03:25.576194 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:25.576161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerStarted","Data":"f26f737e9e6ca6d80667db45c539fce8cc42c5375efc13ae1f183cfe61cdaa25"}
Jul 28 18:03:29.593171 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:29.593085    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerID="77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54" exitCode=0
Jul 28 18:03:29.593651 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:29.593163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerDied","Data":"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"}
Jul 28 18:03:30.599447 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:30.599416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerStarted","Data":"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"}
Jul 28 18:03:30.599927 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:30.599485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerStarted","Data":"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"}
Jul 28 18:03:30.599927 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:30.599705    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:30.621042 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:30.620994    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podStartSLOduration=2.686396948 podStartE2EDuration="6.620980245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934583297s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:30.619336151 +0000 UTC m=+2778.936775038" watchObservedRunningTime="2026-07-28 18:03:30.620980245 +0000 UTC m=+2778.938419131"
Jul 28 18:03:31.604137 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:31.604104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:31.605105 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:31.605080    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:03:32.607911 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:32.607864    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:03:37.612247 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:37.612179    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:03:37.612731 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:37.612647    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:03:47.612616 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:47.612577    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:03:57.612955 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:03:57.612911    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:04:07.612825 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:07.612787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:04:17.613510 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:17.613480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:04:26.794058 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:26.794023    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:04:26.794512 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:26.794275    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" containerID="cri-o://b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8" gracePeriod=30
Jul 28 18:04:26.794512 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:26.794335    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kube-rbac-proxy" containerID="cri-o://ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3" gracePeriod=30
Jul 28 18:04:27.608641 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:27.608601    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 18:04:27.612889 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:27.612858    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:04:27.827718 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:27.827685    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerID="ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3" exitCode=2
Jul 28 18:04:27.828120 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:27.827754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerDied","Data":"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"}
Jul 28 18:04:29.339426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.339403    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:04:29.453835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.453768    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8998\" (UniqueName: \"kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998\") pod \"d9d7f0b9-07e3-4828-8640-ed456904ea93\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") "
Jul 28 18:04:29.453835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.453820    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls\") pod \"d9d7f0b9-07e3-4828-8640-ed456904ea93\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") "
Jul 28 18:04:29.454010 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.453843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert\") pod \"d9d7f0b9-07e3-4828-8640-ed456904ea93\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") "
Jul 28 18:04:29.454010 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.453863    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config\") pod \"d9d7f0b9-07e3-4828-8640-ed456904ea93\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") "
Jul 28 18:04:29.454010 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.453887    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location\") pod \"d9d7f0b9-07e3-4828-8640-ed456904ea93\" (UID: \"d9d7f0b9-07e3-4828-8640-ed456904ea93\") "
Jul 28 18:04:29.454301 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.454264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9d7f0b9-07e3-4828-8640-ed456904ea93" (UID: "d9d7f0b9-07e3-4828-8640-ed456904ea93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:29.454427 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.454301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9d7f0b9-07e3-4828-8640-ed456904ea93" (UID: "d9d7f0b9-07e3-4828-8640-ed456904ea93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:29.455840 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.455810    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998" (OuterVolumeSpecName: "kube-api-access-b8998") pod "d9d7f0b9-07e3-4828-8640-ed456904ea93" (UID: "d9d7f0b9-07e3-4828-8640-ed456904ea93"). InnerVolumeSpecName "kube-api-access-b8998". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:29.455944 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.455848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9d7f0b9-07e3-4828-8640-ed456904ea93" (UID: "d9d7f0b9-07e3-4828-8640-ed456904ea93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:29.466876 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.466845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9d7f0b9-07e3-4828-8640-ed456904ea93" (UID: "d9d7f0b9-07e3-4828-8640-ed456904ea93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:29.555426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.555403    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8998\" (UniqueName: \"kubernetes.io/projected/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-api-access-b8998\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.555426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.555423    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d7f0b9-07e3-4828-8640-ed456904ea93-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.555547 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.555435    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.555547 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.555443    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d7f0b9-07e3-4828-8640-ed456904ea93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.555547 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.555451    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d7f0b9-07e3-4828-8640-ed456904ea93-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:04:29.837881 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.837799    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerID="b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8" exitCode=0
Jul 28 18:04:29.837881 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.837850    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"
Jul 28 18:04:29.838104 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.837890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerDied","Data":"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"}
Jul 28 18:04:29.838104 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.837965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5" event={"ID":"d9d7f0b9-07e3-4828-8640-ed456904ea93","Type":"ContainerDied","Data":"f26f737e9e6ca6d80667db45c539fce8cc42c5375efc13ae1f183cfe61cdaa25"}
Jul 28 18:04:29.838104 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.838006    2561 scope.go:117] "RemoveContainer" containerID="ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"
Jul 28 18:04:29.846559 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.846529    2561 scope.go:117] "RemoveContainer" containerID="b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"
Jul 28 18:04:29.853477 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.853462    2561 scope.go:117] "RemoveContainer" containerID="77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"
Jul 28 18:04:29.860111 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860095    2561 scope.go:117] "RemoveContainer" containerID="ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"
Jul 28 18:04:29.860356 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:04:29.860334    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3\": container with ID starting with ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3 not found: ID does not exist" containerID="ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"
Jul 28 18:04:29.860459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860362    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3"} err="failed to get container status \"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3\": rpc error: code = NotFound desc = could not find container \"ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3\": container with ID starting with ac05377e64a31e5e35cc7f64ce796a64d92fbef57a74499719197b6483d340b3 not found: ID does not exist"
Jul 28 18:04:29.860459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860378    2561 scope.go:117] "RemoveContainer" containerID="b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"
Jul 28 18:04:29.860640 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:04:29.860623    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8\": container with ID starting with b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8 not found: ID does not exist" containerID="b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"
Jul 28 18:04:29.860686 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860646    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8"} err="failed to get container status \"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8\": rpc error: code = NotFound desc = could not find container \"b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8\": container with ID starting with b405271301ea555c188fe282705f6a75c61f40970945ad881b72f3dab5c2a9a8 not found: ID does not exist"
Jul 28 18:04:29.860686 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860661    2561 scope.go:117] "RemoveContainer" containerID="77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"
Jul 28 18:04:29.860880 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:04:29.860864    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54\": container with ID starting with 77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54 not found: ID does not exist" containerID="77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"
Jul 28 18:04:29.860921 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.860885    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54"} err="failed to get container status \"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54\": rpc error: code = NotFound desc = could not find container \"77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54\": container with ID starting with 77f04b1530fcd620f2f807f189c23898a6a1396f1eef186315813d5cbebb9e54 not found: ID does not exist"
Jul 28 18:04:29.870956 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.870936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:04:29.882727 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:29.882702    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5"]
Jul 28 18:04:30.202897 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:04:30.202869    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" path="/var/lib/kubelet/pods/d9d7f0b9-07e3-4828-8640-ed456904ea93/volumes"
Jul 28 18:11:06.353894 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.353822    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354092    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354103    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354121    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="storage-initializer"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354127    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="storage-initializer"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354139    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kube-rbac-proxy"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kube-rbac-proxy"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354189    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kserve-container"
Jul 28 18:11:06.354364 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.354195    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d7f0b9-07e3-4828-8640-ed456904ea93" containerName="kube-rbac-proxy"
Jul 28 18:11:06.357276 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.357261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.360495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.360470    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a0e545-predictor-serving-cert\""
Jul 28 18:11:06.360620 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.360477    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a0e545-dockercfg-7249p\""
Jul 28 18:11:06.361235 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.361215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a0e545\""
Jul 28 18:11:06.361363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.361214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a0e545-kube-rbac-proxy-sar-config\""
Jul 28 18:11:06.361363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.361218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:11:06.361363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.361218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:11:06.361588 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.361573    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:11:06.371606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.371585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:06.447891 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.447864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.448008 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.447900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.448008 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.447927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.448140 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.448040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfzt2\" (UniqueName: \"kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.448140 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.448075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.548670 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.548642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.548799 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.548687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.548799 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.548730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.548799 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.548783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfzt2\" (UniqueName: \"kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.548970 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.548806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.549210 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.549191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.549469 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.549451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.549521 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.549474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.551050 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.551034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.557592 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.557574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfzt2\" (UniqueName: \"kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2\") pod \"isvc-secondary-a0e545-predictor-74b7c6689d-wgr69\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") " pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.666789 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.666766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:06.797058 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:06.797035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:06.799375 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:11:06.799350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac427654_7353_475f_8476_187e99f10e3e.slice/crio-7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747 WatchSource:0}: Error finding container 7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747: Status 404 returned error can't find the container with id 7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747
Jul 28 18:11:07.409133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:07.409095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerStarted","Data":"21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"}
Jul 28 18:11:07.409577 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:07.409161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerStarted","Data":"7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747"}
Jul 28 18:11:09.418236 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:09.418208    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/0.log"
Jul 28 18:11:09.418646 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:09.418245    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac427654-7353-475f-8476-187e99f10e3e" containerID="21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36" exitCode=1
Jul 28 18:11:09.418646 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:09.418332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerDied","Data":"21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"}
Jul 28 18:11:10.424125 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:10.424098    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/0.log"
Jul 28 18:11:10.424574 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:10.424208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerStarted","Data":"3ed1f1f8380c82061325af6165b1d0732af6db8e1f1e256647372d5e6ba8cb6b"}
Jul 28 18:11:13.438144 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.438116    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/1.log"
Jul 28 18:11:13.438615 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.438526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/0.log"
Jul 28 18:11:13.438615 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.438558    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac427654-7353-475f-8476-187e99f10e3e" containerID="3ed1f1f8380c82061325af6165b1d0732af6db8e1f1e256647372d5e6ba8cb6b" exitCode=1
Jul 28 18:11:13.438694 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.438635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerDied","Data":"3ed1f1f8380c82061325af6165b1d0732af6db8e1f1e256647372d5e6ba8cb6b"}
Jul 28 18:11:13.438748 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.438703    2561 scope.go:117] "RemoveContainer" containerID="21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"
Jul 28 18:11:13.439060 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.439040    2561 scope.go:117] "RemoveContainer" containerID="21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"
Jul 28 18:11:13.449655 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:11:13.449616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_kserve-ci-e2e-test(ac427654-7353-475f-8476-187e99f10e3e)\"" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" podUID="ac427654-7353-475f-8476-187e99f10e3e"
Jul 28 18:11:13.449782 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:11:13.449663    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_kserve-ci-e2e-test_ac427654-7353-475f-8476-187e99f10e3e_0 in pod sandbox 7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747 from index: no such id: '21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36'" containerID="21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"
Jul 28 18:11:13.449782 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:13.449709    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_kserve-ci-e2e-test_ac427654-7353-475f-8476-187e99f10e3e_0 in pod sandbox 7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747 from index: no such id: '21999f1bd1ce56eacdd1e2d7eafc75fc4613a16fa627632da7d17cdd714b4c36'"
Jul 28 18:11:14.443694 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:14.443668    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/1.log"
Jul 28 18:11:20.484071 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.484038    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:20.608353 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.608330    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/1.log"
Jul 28 18:11:20.612055 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.612040    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:20.658360 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location\") pod \"ac427654-7353-475f-8476-187e99f10e3e\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") "
Jul 28 18:11:20.658468 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls\") pod \"ac427654-7353-475f-8476-187e99f10e3e\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") "
Jul 28 18:11:20.658468 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658393    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config\") pod \"ac427654-7353-475f-8476-187e99f10e3e\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") "
Jul 28 18:11:20.658468 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658411    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert\") pod \"ac427654-7353-475f-8476-187e99f10e3e\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") "
Jul 28 18:11:20.658468 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfzt2\" (UniqueName: \"kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2\") pod \"ac427654-7353-475f-8476-187e99f10e3e\" (UID: \"ac427654-7353-475f-8476-187e99f10e3e\") "
Jul 28 18:11:20.658669 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac427654-7353-475f-8476-187e99f10e3e" (UID: "ac427654-7353-475f-8476-187e99f10e3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:20.658732 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658707    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac427654-7353-475f-8476-187e99f10e3e" (UID: "ac427654-7353-475f-8476-187e99f10e3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:20.658791 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.658763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac427654-7353-475f-8476-187e99f10e3e" (UID: "ac427654-7353-475f-8476-187e99f10e3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:20.660442 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.660420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2" (OuterVolumeSpecName: "kube-api-access-mfzt2") pod "ac427654-7353-475f-8476-187e99f10e3e" (UID: "ac427654-7353-475f-8476-187e99f10e3e"). InnerVolumeSpecName "kube-api-access-mfzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:11:20.660528 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.660511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac427654-7353-475f-8476-187e99f10e3e" (UID: "ac427654-7353-475f-8476-187e99f10e3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:11:20.759534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.759470    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfzt2\" (UniqueName: \"kubernetes.io/projected/ac427654-7353-475f-8476-187e99f10e3e-kube-api-access-mfzt2\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:11:20.759534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.759493    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427654-7353-475f-8476-187e99f10e3e-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:11:20.759534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.759506    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac427654-7353-475f-8476-187e99f10e3e-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:11:20.759534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.759521    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:11:20.759534 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:20.759535    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac427654-7353-475f-8476-187e99f10e3e-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:11:21.470752 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.470726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_ac427654-7353-475f-8476-187e99f10e3e/storage-initializer/1.log"
Jul 28 18:11:21.470922 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.470815    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"
Jul 28 18:11:21.470922 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.470855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69" event={"ID":"ac427654-7353-475f-8476-187e99f10e3e","Type":"ContainerDied","Data":"7452858e01f00ab5510fd179b1f5ba96897eac1ec59ccb0b639099b4160ed747"}
Jul 28 18:11:21.470997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.470927    2561 scope.go:117] "RemoveContainer" containerID="3ed1f1f8380c82061325af6165b1d0732af6db8e1f1e256647372d5e6ba8cb6b"
Jul 28 18:11:21.511222 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.511195    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:21.515682 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:21.515661    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69"]
Jul 28 18:11:22.203426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:22.203396    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac427654-7353-475f-8476-187e99f10e3e" path="/var/lib/kubelet/pods/ac427654-7353-475f-8476-187e99f10e3e/volumes"
Jul 28 18:11:40.987073 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987037    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:11:40.987566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987373    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.987566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987390    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.987566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987453    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.987566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987522    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.987566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987529    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.987740 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.987577    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac427654-7353-475f-8476-187e99f10e3e" containerName="storage-initializer"
Jul 28 18:11:40.990563 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.990547    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:40.994338 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.994293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:11:40.994462 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.994400    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:11:40.995052 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.995039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:11:40.995484 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.995463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:11:40.995606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.995504    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:11:40.995606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.995472    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:11:40.995606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:40.995596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 18:11:41.006915 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.006892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:11:41.105754 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.105708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.105881 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.105760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.105881 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.105817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.106000 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.105891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4jp\" (UniqueName: \"kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.106000 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.105918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.206859 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.206833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.206956 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.206863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.206956 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.206889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.206956 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.206941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4jp\" (UniqueName: \"kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.207121 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.207031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.207419 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.207396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.207598 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.207578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.207681 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.207665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.209215 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.209196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.218446 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.218420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4jp\" (UniqueName: \"kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-ljhq9\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.300466 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.300410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:11:41.430066 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.429925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:11:41.432706 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:11:41.432684    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf47b959_b971_47d3_8bf4_8a7f9da6347d.slice/crio-9b34bbdf6d935045ee17ea2786d222e563c92a43e5864db7d6a6b37660a95cf6 WatchSource:0}: Error finding container 9b34bbdf6d935045ee17ea2786d222e563c92a43e5864db7d6a6b37660a95cf6: Status 404 returned error can't find the container with id 9b34bbdf6d935045ee17ea2786d222e563c92a43e5864db7d6a6b37660a95cf6
Jul 28 18:11:41.553101 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.553035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerStarted","Data":"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"}
Jul 28 18:11:41.553101 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:41.553092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerStarted","Data":"9b34bbdf6d935045ee17ea2786d222e563c92a43e5864db7d6a6b37660a95cf6"}
Jul 28 18:11:45.576433 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:45.576399    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerID="44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4" exitCode=0
Jul 28 18:11:45.576748 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:45.576472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerDied","Data":"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"}
Jul 28 18:11:45.577772 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:11:45.577755    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:12:08.700021 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:08.699992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerStarted","Data":"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"}
Jul 28 18:12:08.700429 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:08.700050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerStarted","Data":"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"}
Jul 28 18:12:08.700429 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:08.700274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:12:08.723268 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:08.723224    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podStartSLOduration=2.227634631 podStartE2EDuration="28.723210524s" totalImagesPullingTime="22.609314642s" totalInitContainerRuntime="3.886261251s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:08.72128643 +0000 UTC m=+3297.038725317" watchObservedRunningTime="2026-07-28 18:12:08.723210524 +0000 UTC m=+3297.040649411"
Jul 28 18:12:09.703747 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:09.703722    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:12:09.704894 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:09.704867    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:10.707757 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:10.707718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:15.712036 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:15.712007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:12:15.712524 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:15.712428    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:25.713140 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:25.713101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:35.713345 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:35.713251    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:45.713289 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:45.713247    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:12:55.712951 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:12:55.712909    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:13:05.712991 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:05.712954    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:13:15.712602 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:15.712564    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:13:17.198824 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:17.198785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:13:27.199472 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:27.199439    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:13:31.093794 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:31.093763    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:13:31.094233 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:31.093958    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container" containerID="cri-o://516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983" gracePeriod=30
Jul 28 18:13:31.094233 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:31.094033    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kube-rbac-proxy" containerID="cri-o://8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec" gracePeriod=30
Jul 28 18:13:32.028016 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:32.027982    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerID="8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec" exitCode=2
Jul 28 18:13:32.028195 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:32.028049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerDied","Data":"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"}
Jul 28 18:13:35.334367 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.334342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:13:35.501930 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.501868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config\") pod \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") "
Jul 28 18:13:35.501930 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.501900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert\") pod \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") "
Jul 28 18:13:35.502107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.501936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location\") pod \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") "
Jul 28 18:13:35.502107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs4jp\" (UniqueName: \"kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp\") pod \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") "
Jul 28 18:13:35.502221 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls\") pod \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\" (UID: \"cf47b959-b971-47d3-8bf4-8a7f9da6347d\") "
Jul 28 18:13:35.502274 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502214    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf47b959-b971-47d3-8bf4-8a7f9da6347d" (UID: "cf47b959-b971-47d3-8bf4-8a7f9da6347d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:35.502274 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf47b959-b971-47d3-8bf4-8a7f9da6347d" (UID: "cf47b959-b971-47d3-8bf4-8a7f9da6347d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:35.502414 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf47b959-b971-47d3-8bf4-8a7f9da6347d" (UID: "cf47b959-b971-47d3-8bf4-8a7f9da6347d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:35.502414 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502389    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:13:35.502414 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502401    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf47b959-b971-47d3-8bf4-8a7f9da6347d-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:13:35.502414 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.502410    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:13:35.503981 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.503962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf47b959-b971-47d3-8bf4-8a7f9da6347d" (UID: "cf47b959-b971-47d3-8bf4-8a7f9da6347d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:35.504094 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.504078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp" (OuterVolumeSpecName: "kube-api-access-cs4jp") pod "cf47b959-b971-47d3-8bf4-8a7f9da6347d" (UID: "cf47b959-b971-47d3-8bf4-8a7f9da6347d"). InnerVolumeSpecName "kube-api-access-cs4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:13:35.603399 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.603376    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4jp\" (UniqueName: \"kubernetes.io/projected/cf47b959-b971-47d3-8bf4-8a7f9da6347d-kube-api-access-cs4jp\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:13:35.603399 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:35.603395    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf47b959-b971-47d3-8bf4-8a7f9da6347d-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:13:36.048633 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.048601    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerID="516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983" exitCode=0
Jul 28 18:13:36.048805 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.048652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerDied","Data":"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"}
Jul 28 18:13:36.048805 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.048667    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"
Jul 28 18:13:36.048805 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.048704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9" event={"ID":"cf47b959-b971-47d3-8bf4-8a7f9da6347d","Type":"ContainerDied","Data":"9b34bbdf6d935045ee17ea2786d222e563c92a43e5864db7d6a6b37660a95cf6"}
Jul 28 18:13:36.048805 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.048729    2561 scope.go:117] "RemoveContainer" containerID="8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"
Jul 28 18:13:36.056935 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.056912    2561 scope.go:117] "RemoveContainer" containerID="516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"
Jul 28 18:13:36.064326 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.064294    2561 scope.go:117] "RemoveContainer" containerID="44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"
Jul 28 18:13:36.071272 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071256    2561 scope.go:117] "RemoveContainer" containerID="8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"
Jul 28 18:13:36.071505 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:13:36.071488    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec\": container with ID starting with 8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec not found: ID does not exist" containerID="8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"
Jul 28 18:13:36.071567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071513    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec"} err="failed to get container status \"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec\": rpc error: code = NotFound desc = could not find container \"8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec\": container with ID starting with 8611d0a1fd82f4185020bdf7fc6f9842ac4af24827828be93ceff8f5abe54bec not found: ID does not exist"
Jul 28 18:13:36.071567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071530    2561 scope.go:117] "RemoveContainer" containerID="516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"
Jul 28 18:13:36.071761 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:13:36.071746    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983\": container with ID starting with 516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983 not found: ID does not exist" containerID="516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"
Jul 28 18:13:36.071808 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071765    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983"} err="failed to get container status \"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983\": rpc error: code = NotFound desc = could not find container \"516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983\": container with ID starting with 516c845e4109c5baf3b5cd077526abd474a6972dc5ab671dfdf1c6c385880983 not found: ID does not exist"
Jul 28 18:13:36.071808 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071779    2561 scope.go:117] "RemoveContainer" containerID="44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"
Jul 28 18:13:36.071959 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:13:36.071944    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4\": container with ID starting with 44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4 not found: ID does not exist" containerID="44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"
Jul 28 18:13:36.072008 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.071961    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4"} err="failed to get container status \"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4\": rpc error: code = NotFound desc = could not find container \"44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4\": container with ID starting with 44a4ba931eeae5ff4126a8792925e624c63a7c7f23579d3fb4c3ceee636d8dc4 not found: ID does not exist"
Jul 28 18:13:36.078078 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.078058    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:13:36.085387 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.085370    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9"]
Jul 28 18:13:36.203712 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:13:36.203640    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" path="/var/lib/kubelet/pods/cf47b959-b971-47d3-8bf4-8a7f9da6347d/volumes"
Jul 28 18:18:42.998762 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.998723    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999035    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999047    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999057    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kube-rbac-proxy"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999063    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kube-rbac-proxy"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999072    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="storage-initializer"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999078    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="storage-initializer"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999129    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kube-rbac-proxy"
Jul 28 18:18:42.999225 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:42.999138    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf47b959-b971-47d3-8bf4-8a7f9da6347d" containerName="kserve-container"
Jul 28 18:18:43.002211 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.002192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.005413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.005386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 18:18:43.005894 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.005871    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:18:43.006002 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.005877    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:18:43.006002 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.005912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:18:43.006456 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.006434    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:18:43.006564 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.006488    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:18:43.006693 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.006630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:18:43.009460 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.009438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.009554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.009509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.009614 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.009576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64hv\" (UniqueName: \"kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.009614 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.009602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.009717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.009626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.018230 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.018201    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:18:43.110858 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.110815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.110858 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.110862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.110927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64hv\" (UniqueName: \"kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.110953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.111084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111536 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.111504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111673 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.111625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.111735 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.111701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.113536 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.113510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.125630 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.125609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64hv\" (UniqueName: \"kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.312267 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.312178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:43.454772 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:43.454746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:18:43.457267 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:18:43.457234    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41588ba0_9374_4a87_a8a9_32ff3ce5a3f7.slice/crio-a26de0cd78996175a3fae5ab9cfa7e732793edea8d54ce60e601380cca08fe0c WatchSource:0}: Error finding container a26de0cd78996175a3fae5ab9cfa7e732793edea8d54ce60e601380cca08fe0c: Status 404 returned error can't find the container with id a26de0cd78996175a3fae5ab9cfa7e732793edea8d54ce60e601380cca08fe0c
Jul 28 18:18:44.264071 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:44.264034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerStarted","Data":"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"}
Jul 28 18:18:44.264494 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:44.264094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerStarted","Data":"a26de0cd78996175a3fae5ab9cfa7e732793edea8d54ce60e601380cca08fe0c"}
Jul 28 18:18:47.278371 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:47.278333    2561 generic.go:356] "Generic (PLEG): container finished" podID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerID="9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699" exitCode=0
Jul 28 18:18:47.278759 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:47.278402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerDied","Data":"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"}
Jul 28 18:18:48.284997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:48.284960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerStarted","Data":"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"}
Jul 28 18:18:48.285412 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:48.285024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerStarted","Data":"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"}
Jul 28 18:18:48.285412 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:48.285221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:48.305751 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:48.305693    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podStartSLOduration=2.731449357 podStartE2EDuration="6.30567664s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.574227283s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:18:48.305540682 +0000 UTC m=+3696.622979570" watchObservedRunningTime="2026-07-28 18:18:48.30567664 +0000 UTC m=+3696.623115603"
Jul 28 18:18:49.290215 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:49.290185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:18:55.299844 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:18:55.299812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:19:25.301499 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:19:25.301460    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:19:35.301143 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:19:35.301101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:19:45.300325 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:19:45.300269    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:19:55.301087 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:19:55.301044    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:20:05.389556 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:05.389469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:20:13.163666 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:13.163632    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:20:13.164267 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:13.163973    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" containerID="cri-o://7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a" gracePeriod=30
Jul 28 18:20:13.164267 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:13.164049    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kube-rbac-proxy" containerID="cri-o://6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6" gracePeriod=30
Jul 28 18:20:13.628090 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:13.628000    2561 generic.go:356] "Generic (PLEG): container finished" podID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerID="6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6" exitCode=2
Jul 28 18:20:13.628090 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:13.628059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerDied","Data":"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"}
Jul 28 18:20:15.295007 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:15.294967    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 28 18:20:15.300557 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:15.300524    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:20:18.116105 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.116078    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:20:18.218944 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.218918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert\") pod \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") "
Jul 28 18:20:18.219112 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.218965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location\") pod \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") "
Jul 28 18:20:18.219112 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b64hv\" (UniqueName: \"kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv\") pod \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") "
Jul 28 18:20:18.219112 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config\") pod \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") "
Jul 28 18:20:18.219255 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219116    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls\") pod \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\" (UID: \"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7\") "
Jul 28 18:20:18.219392 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" (UID: "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:18.219473 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" (UID: "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:20:18.219473 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.219448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" (UID: "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:20:18.221219 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.221188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" (UID: "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:20:18.221357 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.221219    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv" (OuterVolumeSpecName: "kube-api-access-b64hv") pod "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" (UID: "41588ba0-9374-4a87-a8a9-32ff3ce5a3f7"). InnerVolumeSpecName "kube-api-access-b64hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:20:18.320459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.320424    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:20:18.320459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.320451    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:20:18.320459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.320461    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64hv\" (UniqueName: \"kubernetes.io/projected/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-api-access-b64hv\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:20:18.320459 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.320471    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:20:18.320726 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.320480    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:20:18.651427 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.651336    2561 generic.go:356] "Generic (PLEG): container finished" podID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerID="7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a" exitCode=0
Jul 28 18:20:18.651427 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.651388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerDied","Data":"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"}
Jul 28 18:20:18.651427 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.651411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"
Jul 28 18:20:18.651666 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.651466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr" event={"ID":"41588ba0-9374-4a87-a8a9-32ff3ce5a3f7","Type":"ContainerDied","Data":"a26de0cd78996175a3fae5ab9cfa7e732793edea8d54ce60e601380cca08fe0c"}
Jul 28 18:20:18.651666 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.651492    2561 scope.go:117] "RemoveContainer" containerID="6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"
Jul 28 18:20:18.660916 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.660896    2561 scope.go:117] "RemoveContainer" containerID="7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"
Jul 28 18:20:18.670174 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.670142    2561 scope.go:117] "RemoveContainer" containerID="9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"
Jul 28 18:20:18.676052 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.676028    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:20:18.678630 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.678608    2561 scope.go:117] "RemoveContainer" containerID="6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"
Jul 28 18:20:18.678901 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:20:18.678881    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6\": container with ID starting with 6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6 not found: ID does not exist" containerID="6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"
Jul 28 18:20:18.678954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.678912    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6"} err="failed to get container status \"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6\": rpc error: code = NotFound desc = could not find container \"6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6\": container with ID starting with 6e8f25a1f52851ae3697e829fc923cc77f50d6c80ea40f70ca481d6eb1d849b6 not found: ID does not exist"
Jul 28 18:20:18.678954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.678932    2561 scope.go:117] "RemoveContainer" containerID="7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"
Jul 28 18:20:18.679190 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:20:18.679169    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a\": container with ID starting with 7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a not found: ID does not exist" containerID="7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"
Jul 28 18:20:18.679286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.679195    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a"} err="failed to get container status \"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a\": rpc error: code = NotFound desc = could not find container \"7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a\": container with ID starting with 7c9ebd93177146169d2860fc759254d98e52c0c3b95808c65e30afa7b60b091a not found: ID does not exist"
Jul 28 18:20:18.679286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.679211    2561 scope.go:117] "RemoveContainer" containerID="9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"
Jul 28 18:20:18.679462 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.679436    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr"]
Jul 28 18:20:18.679598 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:20:18.679582    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699\": container with ID starting with 9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699 not found: ID does not exist" containerID="9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"
Jul 28 18:20:18.679637 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:18.679603    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699"} err="failed to get container status \"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699\": rpc error: code = NotFound desc = could not find container \"9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699\": container with ID starting with 9df258d3aca29fe500730afbe846761303a54b5d1500f81269e8826d37354699 not found: ID does not exist"
Jul 28 18:20:20.203046 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:20:20.203014    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" path="/var/lib/kubelet/pods/41588ba0-9374-4a87-a8a9-32ff3ce5a3f7/volumes"
Jul 28 18:24:41.524246 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524163    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524486    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kube-rbac-proxy"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kube-rbac-proxy"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524510    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524515    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524524    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="storage-initializer"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524529    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="storage-initializer"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kserve-container"
Jul 28 18:24:41.524737 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.524607    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41588ba0-9374-4a87-a8a9-32ff3ce5a3f7" containerName="kube-rbac-proxy"
Jul 28 18:24:41.527537 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.527518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.531906 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.531881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:24:41.531906 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.531881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 18:24:41.532148 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.531915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:24:41.532376 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.532359    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:24:41.532452 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.532379    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:24:41.532452 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.532392    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:24:41.532895 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.532883    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:24:41.542599 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.542565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:24:41.606232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.606197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.606470 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.606264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.606470 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.606427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.606470 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.606461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5dh\" (UniqueName: \"kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.606605 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.606480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707263 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707436 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707436 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707436 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5dh\" (UniqueName: \"kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707436 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.707744 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.707725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.708034 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.708010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.708150 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.708119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.709735 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.709719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.716224 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.716199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5dh\" (UniqueName: \"kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-qw8qz\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.837605 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.837513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:41.971967 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:41.971943    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:24:41.974392 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:24:41.974364    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8de693_239f_43c5_bd3c_497d44a48f6e.slice/crio-18cb4455b6fffe1009426229c84172d75da2461fd970ef66639ecf2ebc153959 WatchSource:0}: Error finding container 18cb4455b6fffe1009426229c84172d75da2461fd970ef66639ecf2ebc153959: Status 404 returned error can't find the container with id 18cb4455b6fffe1009426229c84172d75da2461fd970ef66639ecf2ebc153959
Jul 28 18:24:42.684978 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:42.684943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerStarted","Data":"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"}
Jul 28 18:24:42.685417 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:42.685000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerStarted","Data":"18cb4455b6fffe1009426229c84172d75da2461fd970ef66639ecf2ebc153959"}
Jul 28 18:24:47.707426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:47.707392    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerID="60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12" exitCode=0
Jul 28 18:24:47.707818 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:47.707464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerDied","Data":"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"}
Jul 28 18:24:48.713172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.713138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerStarted","Data":"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"}
Jul 28 18:24:48.713589 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.713198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerStarted","Data":"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"}
Jul 28 18:24:48.713589 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.713468    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:48.713691 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.713607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:48.714880 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.714855    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:24:48.736451 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:48.736413    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podStartSLOduration=2.08883981 podStartE2EDuration="7.736397813s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.647558003s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:48.735244274 +0000 UTC m=+4057.052683179" watchObservedRunningTime="2026-07-28 18:24:48.736397813 +0000 UTC m=+4057.053836700"
Jul 28 18:24:49.718577 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:49.718537    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:24:54.723108 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:54.723074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:24:54.723694 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:24:54.723670    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:25:04.724128 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:04.724084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:25:14.370558 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.370526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-599b58cf6-qw8qz_3e8de693-239f-43c5-bd3c-497d44a48f6e/kserve-container/0.log"
Jul 28 18:25:14.539000 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.538968    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:25:14.539247 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.539221    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container" containerID="cri-o://c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac" gracePeriod=30
Jul 28 18:25:14.539332 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.539248    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kube-rbac-proxy" containerID="cri-o://55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e" gracePeriod=30
Jul 28 18:25:14.718865 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.718828    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 28 18:25:14.823098 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.823065    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerID="55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e" exitCode=2
Jul 28 18:25:14.823273 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:14.823120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerDied","Data":"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"}
Jul 28 18:25:15.384322 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.384282    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:25:15.395761 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.395744    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert\") pod \"3e8de693-239f-43c5-bd3c-497d44a48f6e\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") "
Jul 28 18:25:15.395850 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.395775    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location\") pod \"3e8de693-239f-43c5-bd3c-497d44a48f6e\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") "
Jul 28 18:25:15.395850 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.395800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config\") pod \"3e8de693-239f-43c5-bd3c-497d44a48f6e\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") "
Jul 28 18:25:15.395850 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.395817    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5dh\" (UniqueName: \"kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh\") pod \"3e8de693-239f-43c5-bd3c-497d44a48f6e\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") "
Jul 28 18:25:15.395850 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.395837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls\") pod \"3e8de693-239f-43c5-bd3c-497d44a48f6e\" (UID: \"3e8de693-239f-43c5-bd3c-497d44a48f6e\") "
Jul 28 18:25:15.396138 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.396117    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e8de693-239f-43c5-bd3c-497d44a48f6e" (UID: "3e8de693-239f-43c5-bd3c-497d44a48f6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:25:15.396195 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.396130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e8de693-239f-43c5-bd3c-497d44a48f6e" (UID: "3e8de693-239f-43c5-bd3c-497d44a48f6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:25:15.397788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.397759    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh" (OuterVolumeSpecName: "kube-api-access-8z5dh") pod "3e8de693-239f-43c5-bd3c-497d44a48f6e" (UID: "3e8de693-239f-43c5-bd3c-497d44a48f6e"). InnerVolumeSpecName "kube-api-access-8z5dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:25:15.397886 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.397822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e8de693-239f-43c5-bd3c-497d44a48f6e" (UID: "3e8de693-239f-43c5-bd3c-497d44a48f6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:25:15.419863 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.419836    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e8de693-239f-43c5-bd3c-497d44a48f6e" (UID: "3e8de693-239f-43c5-bd3c-497d44a48f6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:25:15.496437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.496348    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:25:15.496437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.496379    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5dh\" (UniqueName: \"kubernetes.io/projected/3e8de693-239f-43c5-bd3c-497d44a48f6e-kube-api-access-8z5dh\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:25:15.496437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.496389    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8de693-239f-43c5-bd3c-497d44a48f6e-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:25:15.496437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.496398    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8de693-239f-43c5-bd3c-497d44a48f6e-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:25:15.496437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.496406    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8de693-239f-43c5-bd3c-497d44a48f6e-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:25:15.828769 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.828685    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerID="c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac" exitCode=0
Jul 28 18:25:15.828769 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.828738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerDied","Data":"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"}
Jul 28 18:25:15.828954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.828802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz" event={"ID":"3e8de693-239f-43c5-bd3c-497d44a48f6e","Type":"ContainerDied","Data":"18cb4455b6fffe1009426229c84172d75da2461fd970ef66639ecf2ebc153959"}
Jul 28 18:25:15.828954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.828829    2561 scope.go:117] "RemoveContainer" containerID="55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"
Jul 28 18:25:15.828954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.828752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"
Jul 28 18:25:15.838087 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.838068    2561 scope.go:117] "RemoveContainer" containerID="c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"
Jul 28 18:25:15.845006 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.844990    2561 scope.go:117] "RemoveContainer" containerID="60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"
Jul 28 18:25:15.852055 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.852033    2561 scope.go:117] "RemoveContainer" containerID="55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"
Jul 28 18:25:15.852492 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:25:15.852416    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e\": container with ID starting with 55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e not found: ID does not exist" containerID="55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"
Jul 28 18:25:15.852492 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.852454    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e"} err="failed to get container status \"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e\": rpc error: code = NotFound desc = could not find container \"55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e\": container with ID starting with 55a6a64b51826ff847ffe9cb9ba5a23a44c7ee6630d7a2bdb5aa3102dc663e5e not found: ID does not exist"
Jul 28 18:25:15.852693 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.852559    2561 scope.go:117] "RemoveContainer" containerID="c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"
Jul 28 18:25:15.852872 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:25:15.852853    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac\": container with ID starting with c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac not found: ID does not exist" containerID="c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"
Jul 28 18:25:15.852955 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.852879    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac"} err="failed to get container status \"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac\": rpc error: code = NotFound desc = could not find container \"c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac\": container with ID starting with c18d53b12d1626d77cce58c4b7006d0e5160226996568c37d3a7f08a5adefbac not found: ID does not exist"
Jul 28 18:25:15.852955 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.852898    2561 scope.go:117] "RemoveContainer" containerID="60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"
Jul 28 18:25:15.853183 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:25:15.853161    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12\": container with ID starting with 60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12 not found: ID does not exist" containerID="60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"
Jul 28 18:25:15.853280 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.853188    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12"} err="failed to get container status \"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12\": rpc error: code = NotFound desc = could not find container \"60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12\": container with ID starting with 60f706d1ae2cf75dd79b436433340f9341e6f1c09e8b6cb654478a84cc32ee12 not found: ID does not exist"
Jul 28 18:25:15.853673 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.853656    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:25:15.857199 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:15.857180    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz"]
Jul 28 18:25:16.203062 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:25:16.203028    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" path="/var/lib/kubelet/pods/3e8de693-239f-43c5-bd3c-497d44a48f6e/volumes"
Jul 28 18:26:24.992746 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.992710    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993020    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993032    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993046    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kube-rbac-proxy"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993052    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kube-rbac-proxy"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993070    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="storage-initializer"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993076    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="storage-initializer"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993163    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kube-rbac-proxy"
Jul 28 18:26:24.993232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.993172    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8de693-239f-43c5-bd3c-497d44a48f6e" containerName="kserve-container"
Jul 28 18:26:24.996891 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.996874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:24.999432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.999408    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:26:24.999616 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.999471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 18:26:24.999616 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.999554    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:26:24.999616 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:24.999593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:26:25.000565 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.000542    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:26:25.000672 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.000577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:26:25.000672 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.000605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:26:25.008702 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.008664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:26:25.076676 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.076645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.076884 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.076685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88gq\" (UniqueName: \"kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.076884 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.076708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.076884 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.076793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.076884 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.076841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.177954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.177914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88gq\" (UniqueName: \"kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.177954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.177956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178210 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178210 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178210 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178523 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178855 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.178943 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.178838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.180529 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.180513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.188387 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.188360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88gq\" (UniqueName: \"kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-z59vd\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.308439 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.308345    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:25.447330 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:25.447282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:26:25.450236 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:26:25.450198    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9189bb_8727_4b1f_b9af_2e852a123e96.slice/crio-54191a757e21413b9422c40e309625e874d030b3ebae90810abf47d38c986f8d WatchSource:0}: Error finding container 54191a757e21413b9422c40e309625e874d030b3ebae90810abf47d38c986f8d: Status 404 returned error can't find the container with id 54191a757e21413b9422c40e309625e874d030b3ebae90810abf47d38c986f8d
Jul 28 18:26:26.109280 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:26.109238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerStarted","Data":"6b574502fd117ef80fd65c03302d74338b8d64e019f3d58b4e22ead4bdfd7d64"}
Jul 28 18:26:26.109739 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:26.109317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerStarted","Data":"54191a757e21413b9422c40e309625e874d030b3ebae90810abf47d38c986f8d"}
Jul 28 18:26:30.127217 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:30.127177    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerID="6b574502fd117ef80fd65c03302d74338b8d64e019f3d58b4e22ead4bdfd7d64" exitCode=0
Jul 28 18:26:30.127711 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:30.127250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerDied","Data":"6b574502fd117ef80fd65c03302d74338b8d64e019f3d58b4e22ead4bdfd7d64"}
Jul 28 18:26:31.132989 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:31.132954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerStarted","Data":"64a288dbbd3d18a10e642f80811dec8019510f52d8372fee8c602239a833cd11"}
Jul 28 18:26:31.133409 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:31.133021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerStarted","Data":"262e420ad5ec33e0d1f8b4b56619a51921551d53cbc264939d72375cd3f5a30f"}
Jul 28 18:26:31.133409 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:31.133222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:31.155377 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:31.155321    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podStartSLOduration=3.527000803 podStartE2EDuration="7.155293528s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628292725s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:26:31.153074675 +0000 UTC m=+4159.470513563" watchObservedRunningTime="2026-07-28 18:26:31.155293528 +0000 UTC m=+4159.472732415"
Jul 28 18:26:32.137532 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:32.137498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:32.138879 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:32.138847    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:26:33.142043 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:33.142006    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:26:38.146448 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:38.146412    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:26:38.146934 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:38.146902    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:26:48.147401 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:48.147360    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:26:58.147176 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:26:58.147128    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:27:08.147550 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:08.147505    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:27:18.147369 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:18.147298    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:27:28.147665 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:28.147626    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:27:38.147593 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:38.147484    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:27:48.148413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:48.148384    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:27:55.223922 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:55.223886    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:27:55.224363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:55.224131    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" containerID="cri-o://262e420ad5ec33e0d1f8b4b56619a51921551d53cbc264939d72375cd3f5a30f" gracePeriod=30
Jul 28 18:27:55.224363 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:55.224178    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kube-rbac-proxy" containerID="cri-o://64a288dbbd3d18a10e642f80811dec8019510f52d8372fee8c602239a833cd11" gracePeriod=30
Jul 28 18:27:55.483273 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:55.483175    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerID="64a288dbbd3d18a10e642f80811dec8019510f52d8372fee8c602239a833cd11" exitCode=2
Jul 28 18:27:55.483273 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:55.483251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerDied","Data":"64a288dbbd3d18a10e642f80811dec8019510f52d8372fee8c602239a833cd11"}
Jul 28 18:27:58.142688 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:58.142636    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 28 18:27:58.147052 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:27:58.147016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:28:00.508530 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.508487    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerID="262e420ad5ec33e0d1f8b4b56619a51921551d53cbc264939d72375cd3f5a30f" exitCode=0
Jul 28 18:28:00.508889 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.508553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerDied","Data":"262e420ad5ec33e0d1f8b4b56619a51921551d53cbc264939d72375cd3f5a30f"}
Jul 28 18:28:00.562024 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.562001    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:28:00.579055 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579019    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls\") pod \"9c9189bb-8727-4b1f-b9af-2e852a123e96\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") "
Jul 28 18:28:00.579152 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert\") pod \"9c9189bb-8727-4b1f-b9af-2e852a123e96\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") "
Jul 28 18:28:00.579203 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config\") pod \"9c9189bb-8727-4b1f-b9af-2e852a123e96\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") "
Jul 28 18:28:00.579240 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location\") pod \"9c9189bb-8727-4b1f-b9af-2e852a123e96\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") "
Jul 28 18:28:00.579287 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j88gq\" (UniqueName: \"kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq\") pod \"9c9189bb-8727-4b1f-b9af-2e852a123e96\" (UID: \"9c9189bb-8727-4b1f-b9af-2e852a123e96\") "
Jul 28 18:28:00.579587 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c9189bb-8727-4b1f-b9af-2e852a123e96" (UID: "9c9189bb-8727-4b1f-b9af-2e852a123e96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:00.579587 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c9189bb-8727-4b1f-b9af-2e852a123e96" (UID: "9c9189bb-8727-4b1f-b9af-2e852a123e96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:00.579753 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.579629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c9189bb-8727-4b1f-b9af-2e852a123e96" (UID: "9c9189bb-8727-4b1f-b9af-2e852a123e96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:00.581263 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.581236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq" (OuterVolumeSpecName: "kube-api-access-j88gq") pod "9c9189bb-8727-4b1f-b9af-2e852a123e96" (UID: "9c9189bb-8727-4b1f-b9af-2e852a123e96"). InnerVolumeSpecName "kube-api-access-j88gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:28:00.581390 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.581294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c9189bb-8727-4b1f-b9af-2e852a123e96" (UID: "9c9189bb-8727-4b1f-b9af-2e852a123e96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:28:00.680366 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.680336    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:28:00.680366 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.680367    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:28:00.680578 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.680380    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9189bb-8727-4b1f-b9af-2e852a123e96-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:28:00.680578 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.680391    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88gq\" (UniqueName: \"kubernetes.io/projected/9c9189bb-8727-4b1f-b9af-2e852a123e96-kube-api-access-j88gq\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:28:00.680578 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:00.680399    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9189bb-8727-4b1f-b9af-2e852a123e96-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:28:01.514554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.514519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd" event={"ID":"9c9189bb-8727-4b1f-b9af-2e852a123e96","Type":"ContainerDied","Data":"54191a757e21413b9422c40e309625e874d030b3ebae90810abf47d38c986f8d"}
Jul 28 18:28:01.515024 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.514562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"
Jul 28 18:28:01.515024 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.514592    2561 scope.go:117] "RemoveContainer" containerID="64a288dbbd3d18a10e642f80811dec8019510f52d8372fee8c602239a833cd11"
Jul 28 18:28:01.523257 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.523238    2561 scope.go:117] "RemoveContainer" containerID="262e420ad5ec33e0d1f8b4b56619a51921551d53cbc264939d72375cd3f5a30f"
Jul 28 18:28:01.531447 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.531423    2561 scope.go:117] "RemoveContainer" containerID="6b574502fd117ef80fd65c03302d74338b8d64e019f3d58b4e22ead4bdfd7d64"
Jul 28 18:28:01.537238 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.537200    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:28:01.541147 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:01.541114    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd"]
Jul 28 18:28:02.204053 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:28:02.204004    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" path="/var/lib/kubelet/pods/9c9189bb-8727-4b1f-b9af-2e852a123e96/volumes"
Jul 28 18:30:22.217392 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217343    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217765    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217783    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217805    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="storage-initializer"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217813    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="storage-initializer"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217828    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kube-rbac-proxy"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217836    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kube-rbac-proxy"
Jul 28 18:30:22.217932 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217929    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kserve-container"
Jul 28 18:30:22.218251 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.217940    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9189bb-8727-4b1f-b9af-2e852a123e96" containerName="kube-rbac-proxy"
Jul 28 18:30:22.221405 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.221377    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.225957 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.225869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 18:30:22.225957 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.225904    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:30:22.226179 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.226107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:30:22.228216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.227532    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:30:22.228216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.227861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:30:22.228216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.227907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:30:22.228216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.228111    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:30:22.241949 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.241919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:30:22.279252 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.279212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rmn\" (UniqueName: \"kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.279481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.279280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.279481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.279328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.279481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.279385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.279481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.279417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.380671 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.380633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.380671 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.380676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.380954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.380699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.380954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.380762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rmn\" (UniqueName: \"kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.380954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.380815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.381058 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:30:22.381033    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 18:30:22.381121 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.381100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.381163 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:30:22.381110    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls podName:3b36a98c-4d99-4cea-94aa-edad688b716a nodeName:}" failed. No retries permitted until 2026-07-28 18:30:22.881092869 +0000 UTC m=+4391.198531749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 28 18:30:22.381519 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.381500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.381562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.381500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.390545 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.390521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rmn\" (UniqueName: \"kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.884542 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.884500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:22.886890 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:22.886867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:23.132720 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:23.132678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:23.288209 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:23.288176    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:30:23.292195 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:30:23.292170    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b36a98c_4d99_4cea_94aa_edad688b716a.slice/crio-08b421f37ffe8578cb8be35d1d2fd565ebe99606ecfad051d2b1e213a2787745 WatchSource:0}: Error finding container 08b421f37ffe8578cb8be35d1d2fd565ebe99606ecfad051d2b1e213a2787745: Status 404 returned error can't find the container with id 08b421f37ffe8578cb8be35d1d2fd565ebe99606ecfad051d2b1e213a2787745
Jul 28 18:30:24.083787 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:24.083745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerStarted","Data":"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"}
Jul 28 18:30:24.084008 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:24.083821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerStarted","Data":"08b421f37ffe8578cb8be35d1d2fd565ebe99606ecfad051d2b1e213a2787745"}
Jul 28 18:30:28.102133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:28.102038    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerID="b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4" exitCode=0
Jul 28 18:30:28.102630 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:28.102118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerDied","Data":"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"}
Jul 28 18:30:28.103525 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:28.103510    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:30:32.123664 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.123582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerStarted","Data":"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"}
Jul 28 18:30:32.123664 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.123643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerStarted","Data":"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"}
Jul 28 18:30:32.124157 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.123906    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:32.124157 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.123950    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:32.125188 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.125165    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:30:32.145968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:32.145920    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podStartSLOduration=1.9877228420000002 podStartE2EDuration="10.145903687s" totalImagesPullingTime="3.736785241s" totalInitContainerRuntime="4.421395604s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:32.144691926 +0000 UTC m=+4400.462130836" watchObservedRunningTime="2026-07-28 18:30:32.145903687 +0000 UTC m=+4400.463342572"
Jul 28 18:30:33.128460 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:33.128416    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:30:38.133466 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:38.133396    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:30:38.133860 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:38.133840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:30:48.135048 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:30:48.135016    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:31:03.372322 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:03.372275    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:31:03.372917 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:03.372548    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container" containerID="cri-o://9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe" gracePeriod=30
Jul 28 18:31:03.372917 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:03.372608    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" containerID="cri-o://13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a" gracePeriod=30
Jul 28 18:31:04.253151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:04.253110    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerID="13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a" exitCode=2
Jul 28 18:31:04.253360 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:04.253157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerDied","Data":"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"}
Jul 28 18:31:08.129214 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:08.129165    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:13.129287 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:13.129237    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:18.129445 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:18.129400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:18.129875 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:18.129568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:31:23.128728 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:23.128679    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:28.128989 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:28.128939    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:33.128667 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:33.128623    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:31:34.028773 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.028744    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:31:34.114615 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.114578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location\") pod \"3b36a98c-4d99-4cea-94aa-edad688b716a\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") "
Jul 28 18:31:34.114800 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.114647    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88rmn\" (UniqueName: \"kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn\") pod \"3b36a98c-4d99-4cea-94aa-edad688b716a\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") "
Jul 28 18:31:34.114800 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.114671    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config\") pod \"3b36a98c-4d99-4cea-94aa-edad688b716a\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") "
Jul 28 18:31:34.114800 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.114710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") pod \"3b36a98c-4d99-4cea-94aa-edad688b716a\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") "
Jul 28 18:31:34.114800 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.114776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert\") pod \"3b36a98c-4d99-4cea-94aa-edad688b716a\" (UID: \"3b36a98c-4d99-4cea-94aa-edad688b716a\") "
Jul 28 18:31:34.115143 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.115105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b36a98c-4d99-4cea-94aa-edad688b716a" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:34.115269 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.115157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b36a98c-4d99-4cea-94aa-edad688b716a" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:34.116785 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.116756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b36a98c-4d99-4cea-94aa-edad688b716a" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:34.116894 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.116879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn" (OuterVolumeSpecName: "kube-api-access-88rmn") pod "3b36a98c-4d99-4cea-94aa-edad688b716a" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a"). InnerVolumeSpecName "kube-api-access-88rmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:34.125893 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.125855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b36a98c-4d99-4cea-94aa-edad688b716a" (UID: "3b36a98c-4d99-4cea-94aa-edad688b716a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:34.216133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.216100    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:31:34.216133 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.216132    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b36a98c-4d99-4cea-94aa-edad688b716a-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:31:34.216543 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.216142    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rmn\" (UniqueName: \"kubernetes.io/projected/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-api-access-88rmn\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:31:34.216543 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.216150    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b36a98c-4d99-4cea-94aa-edad688b716a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:31:34.216543 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.216159    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b36a98c-4d99-4cea-94aa-edad688b716a-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:31:34.369768 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.369738    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerID="9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe" exitCode=137
Jul 28 18:31:34.369938 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.369794    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"
Jul 28 18:31:34.369938 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.369815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerDied","Data":"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"}
Jul 28 18:31:34.369938 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.369884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r" event={"ID":"3b36a98c-4d99-4cea-94aa-edad688b716a","Type":"ContainerDied","Data":"08b421f37ffe8578cb8be35d1d2fd565ebe99606ecfad051d2b1e213a2787745"}
Jul 28 18:31:34.369938 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.369912    2561 scope.go:117] "RemoveContainer" containerID="13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"
Jul 28 18:31:34.378674 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.378498    2561 scope.go:117] "RemoveContainer" containerID="9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"
Jul 28 18:31:34.386606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.386583    2561 scope.go:117] "RemoveContainer" containerID="b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"
Jul 28 18:31:34.388070 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.388048    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:31:34.393314 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.393275    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r"]
Jul 28 18:31:34.395517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.395497    2561 scope.go:117] "RemoveContainer" containerID="13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"
Jul 28 18:31:34.395872 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:31:34.395852    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a\": container with ID starting with 13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a not found: ID does not exist" containerID="13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"
Jul 28 18:31:34.395954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.395886    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a"} err="failed to get container status \"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a\": rpc error: code = NotFound desc = could not find container \"13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a\": container with ID starting with 13b151f6c6ae7089bc3328f3d9f87f30715d93a7c659bc22cb9d45011f7b5d0a not found: ID does not exist"
Jul 28 18:31:34.395954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.395916    2561 scope.go:117] "RemoveContainer" containerID="9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"
Jul 28 18:31:34.396175 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:31:34.396158    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe\": container with ID starting with 9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe not found: ID does not exist" containerID="9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"
Jul 28 18:31:34.396243 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.396183    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe"} err="failed to get container status \"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe\": rpc error: code = NotFound desc = could not find container \"9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe\": container with ID starting with 9e7fdfe176dd53b96b124d97edcaefe292cc996470d63623cc7a60ebec191afe not found: ID does not exist"
Jul 28 18:31:34.396243 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.396207    2561 scope.go:117] "RemoveContainer" containerID="b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"
Jul 28 18:31:34.396497 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:31:34.396479    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4\": container with ID starting with b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4 not found: ID does not exist" containerID="b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"
Jul 28 18:31:34.396551 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:34.396501    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4"} err="failed to get container status \"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4\": rpc error: code = NotFound desc = could not find container \"b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4\": container with ID starting with b601fcc4d4d290ea5566f7e183e61b5b03ee9f41e0ae5323988465913066cac4 not found: ID does not exist"
Jul 28 18:31:36.203764 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:31:36.203730    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" path="/var/lib/kubelet/pods/3b36a98c-4d99-4cea-94aa-edad688b716a/volumes"
Jul 28 18:37:01.059898 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.059812    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060161    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="storage-initializer"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060176    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="storage-initializer"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060185    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060191    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060207    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060276    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kube-rbac-proxy"
Jul 28 18:37:01.060507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.060286    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b36a98c-4d99-4cea-94aa-edad688b716a" containerName="kserve-container"
Jul 28 18:37:01.063433 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.063412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.067128 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.067101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 18:37:01.067286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.067148    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:37:01.067286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.067148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:37:01.067552 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.067539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:37:01.067598 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.067551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:37:01.068409 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.068396    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:37:01.068483 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.068470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:37:01.096640 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.096604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:37:01.132837 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.132794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.133040 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.132854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.133040 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.132934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcc4\" (UniqueName: \"kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.133040 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.133007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.133040 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.133026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234026 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.233982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcc4\" (UniqueName: \"kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234220 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.234067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234220 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.234094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234220 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.234132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234220 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.234166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.234501 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:37:01.234244    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 18:37:01.234501 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:37:01.234356    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls podName:847b53ce-8876-4611-b810-c1bb107fbf9f nodeName:}" failed. No retries permitted until 2026-07-28 18:37:01.73433382 +0000 UTC m=+4790.051772698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-fhs64" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 18:37:01.234665 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.234645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.235028 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.235006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.235067 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.235006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.246641 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.246603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcc4\" (UniqueName: \"kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.739329 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:01.739261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:01.739526 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:37:01.739427    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 18:37:01.739526 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:37:01.739504    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls podName:847b53ce-8876-4611-b810-c1bb107fbf9f nodeName:}" failed. No retries permitted until 2026-07-28 18:37:02.739488712 +0000 UTC m=+4791.056927577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-fhs64" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 18:37:02.747418 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:02.747376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:02.749818 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:02.749796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-fhs64\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:02.873974 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:02.873919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:03.012060 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:03.012033    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:37:03.014119 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:37:03.014095    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847b53ce_8876_4611_b810_c1bb107fbf9f.slice/crio-cbc7e367ae7dfeb7065839cac76af1399879881a921b1b4dff4e4f4fe5fbda6b WatchSource:0}: Error finding container cbc7e367ae7dfeb7065839cac76af1399879881a921b1b4dff4e4f4fe5fbda6b: Status 404 returned error can't find the container with id cbc7e367ae7dfeb7065839cac76af1399879881a921b1b4dff4e4f4fe5fbda6b
Jul 28 18:37:03.650717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:03.650681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerStarted","Data":"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"}
Jul 28 18:37:03.650907 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:03.650741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerStarted","Data":"cbc7e367ae7dfeb7065839cac76af1399879881a921b1b4dff4e4f4fe5fbda6b"}
Jul 28 18:37:07.668053 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:07.668012    2561 generic.go:356] "Generic (PLEG): container finished" podID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerID="0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069" exitCode=0
Jul 28 18:37:07.668532 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:07.668088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerDied","Data":"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"}
Jul 28 18:37:07.669392 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:07.669375    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:37:26.757834 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:26.757795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerStarted","Data":"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"}
Jul 28 18:37:26.758399 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:26.757869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerStarted","Data":"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"}
Jul 28 18:37:26.758399 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:26.758098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:26.779131 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:26.779078    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podStartSLOduration=3.399143087 podStartE2EDuration="25.779062562s" totalImagesPullingTime="18.670888286s" totalInitContainerRuntime="3.709031189s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:26.778826667 +0000 UTC m=+4815.096265556" watchObservedRunningTime="2026-07-28 18:37:26.779062562 +0000 UTC m=+4815.096501448"
Jul 28 18:37:27.762403 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:27.762372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:27.763631 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:27.763608    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:37:28.766411 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:28.766374    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:37:33.771701 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:33.771671    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:37:33.772245 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:33.772216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:37:43.772981 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:43.772942    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:37:53.772257 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:37:53.772217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:38:03.772185 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:03.772099    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:38:13.772199 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:13.772161    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:38:23.772202 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:23.772154    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:38:33.772474 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:33.772441    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:38:41.197059 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:41.197025    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:38:41.197575 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:41.197268    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" containerID="cri-o://f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759" gracePeriod=30
Jul 28 18:38:41.197575 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:41.197319    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kube-rbac-proxy" containerID="cri-o://b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83" gracePeriod=30
Jul 28 18:38:42.063238 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:42.063205    2561 generic.go:356] "Generic (PLEG): container finished" podID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerID="b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83" exitCode=2
Jul 28 18:38:42.063436 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:42.063277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerDied","Data":"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"}
Jul 28 18:38:43.767554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:43.767507    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 28 18:38:43.772972 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:43.772934    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 28 18:38:44.847818 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.847795    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:38:44.917368 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917333    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fcc4\" (UniqueName: \"kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4\") pod \"847b53ce-8876-4611-b810-c1bb107fbf9f\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") "
Jul 28 18:38:44.917566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") pod \"847b53ce-8876-4611-b810-c1bb107fbf9f\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") "
Jul 28 18:38:44.917566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config\") pod \"847b53ce-8876-4611-b810-c1bb107fbf9f\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") "
Jul 28 18:38:44.917566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location\") pod \"847b53ce-8876-4611-b810-c1bb107fbf9f\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") "
Jul 28 18:38:44.917566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert\") pod \"847b53ce-8876-4611-b810-c1bb107fbf9f\" (UID: \"847b53ce-8876-4611-b810-c1bb107fbf9f\") "
Jul 28 18:38:44.917798 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917774    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "847b53ce-8876-4611-b810-c1bb107fbf9f" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:44.917845 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "847b53ce-8876-4611-b810-c1bb107fbf9f" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:38:44.917888 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.917864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "847b53ce-8876-4611-b810-c1bb107fbf9f" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:38:44.919618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.919596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "847b53ce-8876-4611-b810-c1bb107fbf9f" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:38:44.919618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:44.919601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4" (OuterVolumeSpecName: "kube-api-access-8fcc4") pod "847b53ce-8876-4611-b810-c1bb107fbf9f" (UID: "847b53ce-8876-4611-b810-c1bb107fbf9f"). InnerVolumeSpecName "kube-api-access-8fcc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:38:45.018413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.018290    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847b53ce-8876-4611-b810-c1bb107fbf9f-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:38:45.018413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.018353    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:38:45.018413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.018363    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847b53ce-8876-4611-b810-c1bb107fbf9f-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:38:45.018413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.018373    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847b53ce-8876-4611-b810-c1bb107fbf9f-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:38:45.018413 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.018382    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcc4\" (UniqueName: \"kubernetes.io/projected/847b53ce-8876-4611-b810-c1bb107fbf9f-kube-api-access-8fcc4\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:38:45.076353 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.076325    2561 generic.go:356] "Generic (PLEG): container finished" podID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerID="f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759" exitCode=0
Jul 28 18:38:45.076490 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.076384    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"
Jul 28 18:38:45.076490 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.076412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerDied","Data":"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"}
Jul 28 18:38:45.076490 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.076469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64" event={"ID":"847b53ce-8876-4611-b810-c1bb107fbf9f","Type":"ContainerDied","Data":"cbc7e367ae7dfeb7065839cac76af1399879881a921b1b4dff4e4f4fe5fbda6b"}
Jul 28 18:38:45.076611 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.076498    2561 scope.go:117] "RemoveContainer" containerID="b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"
Jul 28 18:38:45.085050 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.085029    2561 scope.go:117] "RemoveContainer" containerID="f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"
Jul 28 18:38:45.092432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.092415    2561 scope.go:117] "RemoveContainer" containerID="0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"
Jul 28 18:38:45.099569 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.099535    2561 scope.go:117] "RemoveContainer" containerID="b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"
Jul 28 18:38:45.099875 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:38:45.099837    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83\": container with ID starting with b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83 not found: ID does not exist" containerID="b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"
Jul 28 18:38:45.099949 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.099876    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83"} err="failed to get container status \"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83\": rpc error: code = NotFound desc = could not find container \"b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83\": container with ID starting with b9bdf28b078d021b9dc936e4116b8626c02a6cc2c46788d1e1a0dd45ad2f7b83 not found: ID does not exist"
Jul 28 18:38:45.099949 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.099901    2561 scope.go:117] "RemoveContainer" containerID="f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"
Jul 28 18:38:45.100155 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:38:45.100140    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759\": container with ID starting with f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759 not found: ID does not exist" containerID="f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"
Jul 28 18:38:45.100202 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.100159    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759"} err="failed to get container status \"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759\": rpc error: code = NotFound desc = could not find container \"f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759\": container with ID starting with f17fd2c27763a9afc410ba03397962d84b39ea5c85a40257b3d1d0390df39759 not found: ID does not exist"
Jul 28 18:38:45.100202 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.100171    2561 scope.go:117] "RemoveContainer" containerID="0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"
Jul 28 18:38:45.100458 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:38:45.100442    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069\": container with ID starting with 0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069 not found: ID does not exist" containerID="0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"
Jul 28 18:38:45.100545 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.100460    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069"} err="failed to get container status \"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069\": rpc error: code = NotFound desc = could not find container \"0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069\": container with ID starting with 0494017630f9a85899e4115f406cd509d25112e3a00ff2bfde253957fa2a6069 not found: ID does not exist"
Jul 28 18:38:45.101392 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.101366    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:38:45.109567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:45.109547    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64"]
Jul 28 18:38:46.205798 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:38:46.205763    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" path="/var/lib/kubelet/pods/847b53ce-8876-4611-b810-c1bb107fbf9f/volumes"
Jul 28 18:39:51.603722 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.603685    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.603977    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.603986    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.603999    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="storage-initializer"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.604006    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="storage-initializer"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.604014    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kube-rbac-proxy"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.604019    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kube-rbac-proxy"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.604071    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kube-rbac-proxy"
Jul 28 18:39:51.604151 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.604080    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="847b53ce-8876-4611-b810-c1bb107fbf9f" containerName="kserve-container"
Jul 28 18:39:51.605981 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.605966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.609266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.609248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:39:51.609626 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.609604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:39:51.609738 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.609639    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:39:51.609738 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.609703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:39:51.609881 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.609855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 18:39:51.611643 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.611614    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:39:51.611745 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.611614    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:39:51.622554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.622528    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:39:51.678354 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.678300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.678354 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.678364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.678618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.678459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxkm\" (UniqueName: \"kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.678618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.678514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.678618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.678559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779651 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.779614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779651 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.779658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779885 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.779684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779885 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.779720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxkm\" (UniqueName: \"kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779885 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.779746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.779885 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:39:51.779790    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:39:51.779885 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:39:51.779874    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls podName:584154ef-e3f9-47d9-ba54-c356cfa2017f nodeName:}" failed. No retries permitted until 2026-07-28 18:39:52.279852692 +0000 UTC m=+4960.597291557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-84dcd" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:39:51.780167 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.780144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.780453 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.780437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.780501 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.780452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:51.788968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:51.788947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxkm\" (UniqueName: \"kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:52.284138 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:52.284094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:52.286528 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:52.286508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-84dcd\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:52.515792 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:52.515753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:52.644625 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:52.644601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:39:52.646587 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:39:52.646559    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584154ef_e3f9_47d9_ba54_c356cfa2017f.slice/crio-68c8c2982fb287333b07cfb2f332cd86a09aaea4c8374f9db74fb65cad261e64 WatchSource:0}: Error finding container 68c8c2982fb287333b07cfb2f332cd86a09aaea4c8374f9db74fb65cad261e64: Status 404 returned error can't find the container with id 68c8c2982fb287333b07cfb2f332cd86a09aaea4c8374f9db74fb65cad261e64
Jul 28 18:39:53.335160 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:53.335128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerStarted","Data":"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"}
Jul 28 18:39:53.335378 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:53.335182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerStarted","Data":"68c8c2982fb287333b07cfb2f332cd86a09aaea4c8374f9db74fb65cad261e64"}
Jul 28 18:39:57.351735 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:57.351702    2561 generic.go:356] "Generic (PLEG): container finished" podID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerID="9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce" exitCode=0
Jul 28 18:39:57.352134 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:57.351776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerDied","Data":"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"}
Jul 28 18:39:58.358142 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.358111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerStarted","Data":"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"}
Jul 28 18:39:58.358610 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.358171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerStarted","Data":"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"}
Jul 28 18:39:58.358610 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.358445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:58.358610 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.358596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:39:58.359670 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.359646    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:39:58.381507 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:58.381448    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podStartSLOduration=3.669459804 podStartE2EDuration="7.381434646s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711974842s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:58.379970197 +0000 UTC m=+4966.697409093" watchObservedRunningTime="2026-07-28 18:39:58.381434646 +0000 UTC m=+4966.698873531"
Jul 28 18:39:59.362091 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:39:59.362056    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:04.366479 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:04.366452    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:40:04.367135 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:04.367107    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:14.367337 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:14.367288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:24.367190 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:24.367152    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:34.367342 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:34.367287    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:44.367357 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:44.367296    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:40:54.367056 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:40:54.367016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:41:04.367513 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:04.367437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:41:11.704733 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:11.704696    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:41:11.705264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:11.704969    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" containerID="cri-o://6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0" gracePeriod=30
Jul 28 18:41:11.705264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:11.705016    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kube-rbac-proxy" containerID="cri-o://274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38" gracePeriod=30
Jul 28 18:41:12.644154 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:12.644117    2561 generic.go:356] "Generic (PLEG): container finished" podID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerID="274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38" exitCode=2
Jul 28 18:41:12.644353 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:12.644189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerDied","Data":"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"}
Jul 28 18:41:14.362300 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:14.362255    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 28 18:41:14.367888 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:14.367856    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 28 18:41:15.453109 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.453086    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:41:15.489742 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.489712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkxkm\" (UniqueName: \"kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm\") pod \"584154ef-e3f9-47d9-ba54-c356cfa2017f\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") "
Jul 28 18:41:15.489940 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.489764    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location\") pod \"584154ef-e3f9-47d9-ba54-c356cfa2017f\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") "
Jul 28 18:41:15.489940 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.489800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") pod \"584154ef-e3f9-47d9-ba54-c356cfa2017f\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") "
Jul 28 18:41:15.489940 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.489817    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert\") pod \"584154ef-e3f9-47d9-ba54-c356cfa2017f\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") "
Jul 28 18:41:15.489940 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.489843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config\") pod \"584154ef-e3f9-47d9-ba54-c356cfa2017f\" (UID: \"584154ef-e3f9-47d9-ba54-c356cfa2017f\") "
Jul 28 18:41:15.490291 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.490257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "584154ef-e3f9-47d9-ba54-c356cfa2017f" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:41:15.490291 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.490275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "584154ef-e3f9-47d9-ba54-c356cfa2017f" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:41:15.490463 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.490388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "584154ef-e3f9-47d9-ba54-c356cfa2017f" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:41:15.491885 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.491864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "584154ef-e3f9-47d9-ba54-c356cfa2017f" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:41:15.492024 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.492005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm" (OuterVolumeSpecName: "kube-api-access-qkxkm") pod "584154ef-e3f9-47d9-ba54-c356cfa2017f" (UID: "584154ef-e3f9-47d9-ba54-c356cfa2017f"). InnerVolumeSpecName "kube-api-access-qkxkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:41:15.590541 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.590457    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxkm\" (UniqueName: \"kubernetes.io/projected/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-api-access-qkxkm\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:41:15.590541 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.590484    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/584154ef-e3f9-47d9-ba54-c356cfa2017f-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:41:15.590541 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.590493    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/584154ef-e3f9-47d9-ba54-c356cfa2017f-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:41:15.590541 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.590503    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:41:15.590541 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.590511    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/584154ef-e3f9-47d9-ba54-c356cfa2017f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:41:15.656684 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.656645    2561 generic.go:356] "Generic (PLEG): container finished" podID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerID="6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0" exitCode=0
Jul 28 18:41:15.656867 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.656703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"
Jul 28 18:41:15.656867 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.656732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerDied","Data":"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"}
Jul 28 18:41:15.656867 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.656799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd" event={"ID":"584154ef-e3f9-47d9-ba54-c356cfa2017f","Type":"ContainerDied","Data":"68c8c2982fb287333b07cfb2f332cd86a09aaea4c8374f9db74fb65cad261e64"}
Jul 28 18:41:15.656867 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.656830    2561 scope.go:117] "RemoveContainer" containerID="274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"
Jul 28 18:41:15.665721 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.665705    2561 scope.go:117] "RemoveContainer" containerID="6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"
Jul 28 18:41:15.672934 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.672917    2561 scope.go:117] "RemoveContainer" containerID="9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"
Jul 28 18:41:15.679004 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.678982    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:41:15.680142 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.680128    2561 scope.go:117] "RemoveContainer" containerID="274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"
Jul 28 18:41:15.680442 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:41:15.680422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38\": container with ID starting with 274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38 not found: ID does not exist" containerID="274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"
Jul 28 18:41:15.680519 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.680456    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38"} err="failed to get container status \"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38\": rpc error: code = NotFound desc = could not find container \"274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38\": container with ID starting with 274b733416953247a1e9cdda081c01ea438af96d4d0bef8e4b49fa79bf557a38 not found: ID does not exist"
Jul 28 18:41:15.680519 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.680480    2561 scope.go:117] "RemoveContainer" containerID="6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"
Jul 28 18:41:15.680827 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:41:15.680754    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0\": container with ID starting with 6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0 not found: ID does not exist" containerID="6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"
Jul 28 18:41:15.680827 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.680792    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0"} err="failed to get container status \"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0\": rpc error: code = NotFound desc = could not find container \"6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0\": container with ID starting with 6caad071d50f57b4fe8cfb3690868e60f52184a6036fdbb53198d27bb25627d0 not found: ID does not exist"
Jul 28 18:41:15.680827 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.680822    2561 scope.go:117] "RemoveContainer" containerID="9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"
Jul 28 18:41:15.681260 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:41:15.681170    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce\": container with ID starting with 9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce not found: ID does not exist" containerID="9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"
Jul 28 18:41:15.681260 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.681209    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce"} err="failed to get container status \"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce\": rpc error: code = NotFound desc = could not find container \"9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce\": container with ID starting with 9f70d6c7f23e68ede944f919a543a1f76b7e36d15fbbf84e0907b9195fc92dce not found: ID does not exist"
Jul 28 18:41:15.682865 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:15.682848    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd"]
Jul 28 18:41:16.202828 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.202796    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" path="/var/lib/kubelet/pods/584154ef-e3f9-47d9-ba54-c356cfa2017f/volumes"
Jul 28 18:41:16.894747 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.894707    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895031    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kube-rbac-proxy"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895043    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kube-rbac-proxy"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895059    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="storage-initializer"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895065    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="storage-initializer"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895071    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895077    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895128    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kserve-container"
Jul 28 18:41:16.895132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.895137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="584154ef-e3f9-47d9-ba54-c356cfa2017f" containerName="kube-rbac-proxy"
Jul 28 18:41:16.897919 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.897902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:16.900437 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.900417    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:41:16.900558 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.900544    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:41:16.900631 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.900556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:41:16.900631 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.900559    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:41:16.901266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.901253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 18:41:16.901326 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.901295    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 18:41:16.908912 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:16.908891    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:41:17.001265 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.001226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.001503 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.001278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5q2\" (UniqueName: \"kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.001503 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.001382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.001503 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.001433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.102352 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.102284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.102562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.102363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5q2\" (UniqueName: \"kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.102562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.102392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.102562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.102417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.103065 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.103042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.103181 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.103152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.104994 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.104967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.111590 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.111568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5q2\" (UniqueName: \"kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-cl7js\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.207701 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.207673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:17.336631 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.336479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:41:17.339302 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:41:17.339279    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4bc633_3395_4412_8813_a256ef58d96b.slice/crio-3f2682d8144f4c2e9fa82788c55c727ee25a39de7332796fbf01b1ee4b9befc8 WatchSource:0}: Error finding container 3f2682d8144f4c2e9fa82788c55c727ee25a39de7332796fbf01b1ee4b9befc8: Status 404 returned error can't find the container with id 3f2682d8144f4c2e9fa82788c55c727ee25a39de7332796fbf01b1ee4b9befc8
Jul 28 18:41:17.669316 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.669264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerStarted","Data":"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"}
Jul 28 18:41:17.669525 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:17.669348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerStarted","Data":"3f2682d8144f4c2e9fa82788c55c727ee25a39de7332796fbf01b1ee4b9befc8"}
Jul 28 18:41:18.674679 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:18.674640    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d4bc633-3395-4412-8813-a256ef58d96b" containerID="321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54" exitCode=0
Jul 28 18:41:18.675029 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:18.674703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerDied","Data":"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"}
Jul 28 18:41:19.680852 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:19.680815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerStarted","Data":"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"}
Jul 28 18:41:19.681237 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:19.680873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerStarted","Data":"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"}
Jul 28 18:41:19.681237 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:19.680933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:19.702414 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:19.702358    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podStartSLOduration=2.784305174 podStartE2EDuration="3.702328085s" totalImagesPullingTime="0s" totalInitContainerRuntime="918.022911ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:19.701546691 +0000 UTC m=+5048.018985582" watchObservedRunningTime="2026-07-28 18:41:19.702328085 +0000 UTC m=+5048.019766964"
Jul 28 18:41:20.685297 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:20.685267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:20.686453 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:20.686418    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:41:21.689706 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:21.689672    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:41:26.694351 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:26.694296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:41:26.696777 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:26.694799    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:41:36.694876 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:36.694834    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:41:46.695378 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:46.695337    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:41:56.695655 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:41:56.695615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:42:06.695128 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:06.695080    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:42:16.694950 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:16.694912    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:42:26.695411 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:26.695371    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:42:33.200139 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:33.200107    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:42:36.985872 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:36.985839    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:42:36.986282 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:36.986080    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container" containerID="cri-o://f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76" gracePeriod=30
Jul 28 18:42:36.986282 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:36.986149    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kube-rbac-proxy" containerID="cri-o://41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a" gracePeriod=30
Jul 28 18:42:37.140864 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.140822    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:42:37.144453 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.144428    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.147341 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.147317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 18:42:37.147341 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.147337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:42:37.147530 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.147340    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:42:37.155709 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.155685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:42:37.296105 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.296016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.296105 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.296070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.296295 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.296124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.296295 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.296177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnrb\" (UniqueName: \"kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.296295 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.296200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397397 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnrb\" (UniqueName: \"kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397397 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397620 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397620 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397698 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.397818 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.397799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.398163 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.398138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.398258 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.398175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.400162 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.400138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.406420 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.406396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnrb\" (UniqueName: \"kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.454835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.454798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:37.585315 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.585281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:42:37.587754 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:42:37.587731    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403242bb_a2a6_41c8_9fc6_f1899a9abf61.slice/crio-54334b76b6ff1636f626a43e9aeea514b3b5599b0c26468f595c4c03fc08ef1a WatchSource:0}: Error finding container 54334b76b6ff1636f626a43e9aeea514b3b5599b0c26468f595c4c03fc08ef1a: Status 404 returned error can't find the container with id 54334b76b6ff1636f626a43e9aeea514b3b5599b0c26468f595c4c03fc08ef1a
Jul 28 18:42:37.999273 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.999215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerStarted","Data":"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"}
Jul 28 18:42:37.999745 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:37.999340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerStarted","Data":"54334b76b6ff1636f626a43e9aeea514b3b5599b0c26468f595c4c03fc08ef1a"}
Jul 28 18:42:38.004809 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:38.004774    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d4bc633-3395-4412-8813-a256ef58d96b" containerID="41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a" exitCode=2
Jul 28 18:42:38.004966 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:38.004833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerDied","Data":"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"}
Jul 28 18:42:39.009721 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:39.009686    2561 generic.go:356] "Generic (PLEG): container finished" podID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerID="c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46" exitCode=0
Jul 28 18:42:39.010104 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:39.009758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerDied","Data":"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"}
Jul 28 18:42:40.015340 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:40.015286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerStarted","Data":"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"}
Jul 28 18:42:40.015747 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:40.015373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerStarted","Data":"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"}
Jul 28 18:42:40.015747 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:40.015408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:40.036324 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:40.036261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podStartSLOduration=2.093390757 podStartE2EDuration="3.036244848s" totalImagesPullingTime="0s" totalInitContainerRuntime="942.854091ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:42:40.034502787 +0000 UTC m=+5128.351941692" watchObservedRunningTime="2026-07-28 18:42:40.036244848 +0000 UTC m=+5128.353683809"
Jul 28 18:42:41.019656 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.019620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:41.020945 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.020917    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:42:41.690809 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.690773    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Jul 28 18:42:41.846727 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.846702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:42:41.943170 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls\") pod \"2d4bc633-3395-4412-8813-a256ef58d96b\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") "
Jul 28 18:42:41.943170 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location\") pod \"2d4bc633-3395-4412-8813-a256ef58d96b\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") "
Jul 28 18:42:41.943400 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config\") pod \"2d4bc633-3395-4412-8813-a256ef58d96b\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") "
Jul 28 18:42:41.943400 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb5q2\" (UniqueName: \"kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2\") pod \"2d4bc633-3395-4412-8813-a256ef58d96b\" (UID: \"2d4bc633-3395-4412-8813-a256ef58d96b\") "
Jul 28 18:42:41.943562 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d4bc633-3395-4412-8813-a256ef58d96b" (UID: "2d4bc633-3395-4412-8813-a256ef58d96b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:41.943621 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.943542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d4bc633-3395-4412-8813-a256ef58d96b" (UID: "2d4bc633-3395-4412-8813-a256ef58d96b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:42:41.945219 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.945196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d4bc633-3395-4412-8813-a256ef58d96b" (UID: "2d4bc633-3395-4412-8813-a256ef58d96b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:42:41.945334 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:41.945254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2" (OuterVolumeSpecName: "kube-api-access-xb5q2") pod "2d4bc633-3395-4412-8813-a256ef58d96b" (UID: "2d4bc633-3395-4412-8813-a256ef58d96b"). InnerVolumeSpecName "kube-api-access-xb5q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:42.025111 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025080    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d4bc633-3395-4412-8813-a256ef58d96b" containerID="f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76" exitCode=0
Jul 28 18:42:42.025552 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025136    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"
Jul 28 18:42:42.025552 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerDied","Data":"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"}
Jul 28 18:42:42.025552 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js" event={"ID":"2d4bc633-3395-4412-8813-a256ef58d96b","Type":"ContainerDied","Data":"3f2682d8144f4c2e9fa82788c55c727ee25a39de7332796fbf01b1ee4b9befc8"}
Jul 28 18:42:42.025552 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025228    2561 scope.go:117] "RemoveContainer" containerID="41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"
Jul 28 18:42:42.025893 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.025867    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:42:42.033786 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.033764    2561 scope.go:117] "RemoveContainer" containerID="f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"
Jul 28 18:42:42.040853 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.040833    2561 scope.go:117] "RemoveContainer" containerID="321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"
Jul 28 18:42:42.044477 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.044455    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d4bc633-3395-4412-8813-a256ef58d96b-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:42:42.044571 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.044479    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d4bc633-3395-4412-8813-a256ef58d96b-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:42:42.044571 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.044489    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d4bc633-3395-4412-8813-a256ef58d96b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:42:42.044571 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.044498    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5q2\" (UniqueName: \"kubernetes.io/projected/2d4bc633-3395-4412-8813-a256ef58d96b-kube-api-access-xb5q2\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:42:42.048968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.048942    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:42:42.049792 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.049770    2561 scope.go:117] "RemoveContainer" containerID="41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"
Jul 28 18:42:42.055492 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:42:42.055228    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a\": container with ID starting with 41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a not found: ID does not exist" containerID="41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"
Jul 28 18:42:42.055638 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.055507    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a"} err="failed to get container status \"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a\": rpc error: code = NotFound desc = could not find container \"41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a\": container with ID starting with 41355079a1e306c0dac413f1f717f1af3e2aa363cdc5b28b93521a64d16b447a not found: ID does not exist"
Jul 28 18:42:42.055638 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.055535    2561 scope.go:117] "RemoveContainer" containerID="f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"
Jul 28 18:42:42.055918 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:42:42.055895    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76\": container with ID starting with f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76 not found: ID does not exist" containerID="f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"
Jul 28 18:42:42.055983 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.055926    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76"} err="failed to get container status \"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76\": rpc error: code = NotFound desc = could not find container \"f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76\": container with ID starting with f9478b0b29208ab8c73cc1666990a82c1f9546438a227ebd9fc5898480b17a76 not found: ID does not exist"
Jul 28 18:42:42.055983 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.055942    2561 scope.go:117] "RemoveContainer" containerID="321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"
Jul 28 18:42:42.056202 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:42:42.056182    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54\": container with ID starting with 321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54 not found: ID does not exist" containerID="321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"
Jul 28 18:42:42.056253 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.056203    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54"} err="failed to get container status \"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54\": rpc error: code = NotFound desc = could not find container \"321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54\": container with ID starting with 321b7f83884172c7a5bfc9930325e97f9304a97955a1b7cdd5135c5cf7c86b54 not found: ID does not exist"
Jul 28 18:42:42.057405 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.057382    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js"]
Jul 28 18:42:42.203244 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:42.203160    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" path="/var/lib/kubelet/pods/2d4bc633-3395-4412-8813-a256ef58d96b/volumes"
Jul 28 18:42:47.029933 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:47.029905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:42:47.030573 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:47.030544    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:42:57.031212 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:42:57.031169    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:07.030910 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:07.030867    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:17.031216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:17.031173    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:27.030793 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:27.030748    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:37.030526 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:37.030480    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:47.030856 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:47.030809    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:43:57.031463 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:43:57.031434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:44:07.213299 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:07.213261    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:44:07.213717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:07.213566    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container" containerID="cri-o://26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc" gracePeriod=30
Jul 28 18:44:07.213717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:07.213618    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kube-rbac-proxy" containerID="cri-o://5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203" gracePeriod=30
Jul 28 18:44:07.363351 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:07.363289    2561 generic.go:356] "Generic (PLEG): container finished" podID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerID="5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203" exitCode=2
Jul 28 18:44:07.363520 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:07.363362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerDied","Data":"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"}
Jul 28 18:44:11.966154 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:11.966130    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:44:12.086826 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.086745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location\") pod \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") "
Jul 28 18:44:12.086826 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.086781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config\") pod \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") "
Jul 28 18:44:12.086826 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.086805    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnrb\" (UniqueName: \"kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb\") pod \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") "
Jul 28 18:44:12.087126 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.086874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls\") pod \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") "
Jul 28 18:44:12.087126 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.086894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert\") pod \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\" (UID: \"403242bb-a2a6-41c8-9fc6-f1899a9abf61\") "
Jul 28 18:44:12.087232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.087166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "403242bb-a2a6-41c8-9fc6-f1899a9abf61" (UID: "403242bb-a2a6-41c8-9fc6-f1899a9abf61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:44:12.087232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.087185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "403242bb-a2a6-41c8-9fc6-f1899a9abf61" (UID: "403242bb-a2a6-41c8-9fc6-f1899a9abf61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:44:12.087232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.087222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "403242bb-a2a6-41c8-9fc6-f1899a9abf61" (UID: "403242bb-a2a6-41c8-9fc6-f1899a9abf61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:44:12.088906 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.088884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "403242bb-a2a6-41c8-9fc6-f1899a9abf61" (UID: "403242bb-a2a6-41c8-9fc6-f1899a9abf61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:44:12.088983 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.088951    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb" (OuterVolumeSpecName: "kube-api-access-jrnrb") pod "403242bb-a2a6-41c8-9fc6-f1899a9abf61" (UID: "403242bb-a2a6-41c8-9fc6-f1899a9abf61"). InnerVolumeSpecName "kube-api-access-jrnrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:44:12.187654 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.187625    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403242bb-a2a6-41c8-9fc6-f1899a9abf61-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:44:12.187654 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.187649    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:44:12.187654 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.187658    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:44:12.187900 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.187667    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:44:12.187900 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.187676    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnrb\" (UniqueName: \"kubernetes.io/projected/403242bb-a2a6-41c8-9fc6-f1899a9abf61-kube-api-access-jrnrb\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:44:12.388017 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.387926    2561 generic.go:356] "Generic (PLEG): container finished" podID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerID="26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc" exitCode=0
Jul 28 18:44:12.388017 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.387996    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"
Jul 28 18:44:12.388017 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.388004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerDied","Data":"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"}
Jul 28 18:44:12.388296 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.388066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658" event={"ID":"403242bb-a2a6-41c8-9fc6-f1899a9abf61","Type":"ContainerDied","Data":"54334b76b6ff1636f626a43e9aeea514b3b5599b0c26468f595c4c03fc08ef1a"}
Jul 28 18:44:12.388296 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.388092    2561 scope.go:117] "RemoveContainer" containerID="5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"
Jul 28 18:44:12.396128 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.396112    2561 scope.go:117] "RemoveContainer" containerID="26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"
Jul 28 18:44:12.403289 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.403251    2561 scope.go:117] "RemoveContainer" containerID="c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"
Jul 28 18:44:12.404847 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.404831    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:44:12.407439 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.407418    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658"]
Jul 28 18:44:12.411453 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.411438    2561 scope.go:117] "RemoveContainer" containerID="5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"
Jul 28 18:44:12.411749 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:44:12.411728    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203\": container with ID starting with 5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203 not found: ID does not exist" containerID="5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"
Jul 28 18:44:12.411830 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.411757    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203"} err="failed to get container status \"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203\": rpc error: code = NotFound desc = could not find container \"5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203\": container with ID starting with 5f5946cd5e8085717db91ffebfff2ac0f967e27a2f449abf61d0e0ee8270f203 not found: ID does not exist"
Jul 28 18:44:12.411830 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.411774    2561 scope.go:117] "RemoveContainer" containerID="26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"
Jul 28 18:44:12.411992 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:44:12.411974    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc\": container with ID starting with 26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc not found: ID does not exist" containerID="26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"
Jul 28 18:44:12.412055 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.412001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc"} err="failed to get container status \"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc\": rpc error: code = NotFound desc = could not find container \"26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc\": container with ID starting with 26ea8cff58c793c83f2d382a666350d243de5ed306e649a9466ad48349214bcc not found: ID does not exist"
Jul 28 18:44:12.412055 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.412024    2561 scope.go:117] "RemoveContainer" containerID="c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"
Jul 28 18:44:12.412255 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:44:12.412234    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46\": container with ID starting with c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46 not found: ID does not exist" containerID="c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"
Jul 28 18:44:12.412293 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:12.412259    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46"} err="failed to get container status \"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46\": rpc error: code = NotFound desc = could not find container \"c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46\": container with ID starting with c284aca57b0788c7759dcf07a0ed4ce88f05a000120d509ad6590d09351dfa46 not found: ID does not exist"
Jul 28 18:44:14.203899 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:14.203865    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" path="/var/lib/kubelet/pods/403242bb-a2a6-41c8-9fc6-f1899a9abf61/volumes"
Jul 28 18:44:19.458583 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458551    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458858    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458870    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458882    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458887    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458894    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="storage-initializer"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="storage-initializer"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458910    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458915    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458920    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458925    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458934    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="storage-initializer"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.458939    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="storage-initializer"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.459005    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kserve-container"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.459015    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.459021    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4bc633-3395-4412-8813-a256ef58d96b" containerName="kube-rbac-proxy"
Jul 28 18:44:19.459054 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.459029    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="403242bb-a2a6-41c8-9fc6-f1899a9abf61" containerName="kserve-container"
Jul 28 18:44:19.463428 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.463412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.466692 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.466672    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:44:19.466935 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.466921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 18:44:19.467275 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.467258    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:44:19.467485 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.467466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:44:19.467485 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.467479    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:44:19.467485 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.467466    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:44:19.467866 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.467849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:44:19.477580 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.477552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:44:19.550324 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.550266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.550517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.550341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mvq\" (UniqueName: \"kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.550517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.550387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.550517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.550412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.550517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.550463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.650903 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.650863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.650903 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.650906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mvq\" (UniqueName: \"kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651379 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651608 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651585    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.651787 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.651768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.653460 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.653443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.660871 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.660849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mvq\" (UniqueName: \"kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.778252 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.778159    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:19.910497 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:19.910471    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:44:19.912322 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:44:19.912288    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ea1906_fe1b_41a9_bcb3_7e3aa22486e0.slice/crio-4467d9a1c08e2c43948907bbf509c458e89ece52228091e625fdb2bab82fb3c9 WatchSource:0}: Error finding container 4467d9a1c08e2c43948907bbf509c458e89ece52228091e625fdb2bab82fb3c9: Status 404 returned error can't find the container with id 4467d9a1c08e2c43948907bbf509c458e89ece52228091e625fdb2bab82fb3c9
Jul 28 18:44:20.421539 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:20.421499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerStarted","Data":"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"}
Jul 28 18:44:20.421746 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:20.421559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerStarted","Data":"4467d9a1c08e2c43948907bbf509c458e89ece52228091e625fdb2bab82fb3c9"}
Jul 28 18:44:21.426749 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:21.426708    2561 generic.go:356] "Generic (PLEG): container finished" podID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerID="14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1" exitCode=0
Jul 28 18:44:21.427176 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:21.426777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerDied","Data":"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"}
Jul 28 18:44:22.433438 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:22.433407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerStarted","Data":"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"}
Jul 28 18:44:22.433835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:22.433464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerStarted","Data":"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"}
Jul 28 18:44:22.433835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:22.433533    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:22.455780 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:22.455734    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podStartSLOduration=2.515254343 podStartE2EDuration="3.455718374s" totalImagesPullingTime="0s" totalInitContainerRuntime="940.464031ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:44:22.45440532 +0000 UTC m=+5230.771844224" watchObservedRunningTime="2026-07-28 18:44:22.455718374 +0000 UTC m=+5230.773157259"
Jul 28 18:44:23.437649 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:23.437613    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:23.438605 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:23.438582    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:44:24.441602 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:24.441556    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:44:29.445692 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:29.445664    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:44:29.446206 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:29.446180    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:44:39.446876 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:39.446839    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:44:49.446114 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:49.446073    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:44:59.446603 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:44:59.446562    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:45:09.446669 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:09.446632    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:45:19.446868 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:19.446824    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:45:29.446537 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:29.446495    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:45:39.447331 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:39.447236    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:45:49.498928 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:49.498893    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:45:49.499482 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:49.499177    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container" containerID="cri-o://e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1" gracePeriod=30
Jul 28 18:45:49.499482 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:49.499198    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kube-rbac-proxy" containerID="cri-o://aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24" gracePeriod=30
Jul 28 18:45:49.780376 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:49.780278    2561 generic.go:356] "Generic (PLEG): container finished" podID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerID="aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24" exitCode=2
Jul 28 18:45:49.780376 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:49.780351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerDied","Data":"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"}
Jul 28 18:45:50.627715 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.627678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:45:50.630891 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.630872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.633134 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.633107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 18:45:50.633252 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.633235    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:45:50.640434 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.640413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:45:50.680481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.680453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.680481 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.680481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.680647 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.680509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgks\" (UniqueName: \"kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.680647 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.680578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.780975 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.780944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.781141 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.781012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.781141 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.781031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.781141 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.781057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgks\" (UniqueName: \"kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.781554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.781527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.781835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.781813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.783443 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.783425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.788946 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.788887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgks\" (UniqueName: \"kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:50.940508 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:50.940462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:45:51.081346 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:51.081299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:45:51.084379 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:45:51.084338    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679b596e_17b3_4ee1_89ec_8f1466f45533.slice/crio-66a98e8b74711a4c66d9978aa74e7fad73cb0a80faff8e9c0e7c58d296167fb8 WatchSource:0}: Error finding container 66a98e8b74711a4c66d9978aa74e7fad73cb0a80faff8e9c0e7c58d296167fb8: Status 404 returned error can't find the container with id 66a98e8b74711a4c66d9978aa74e7fad73cb0a80faff8e9c0e7c58d296167fb8
Jul 28 18:45:51.790738 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:51.790707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerStarted","Data":"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"}
Jul 28 18:45:51.791155 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:51.790771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerStarted","Data":"66a98e8b74711a4c66d9978aa74e7fad73cb0a80faff8e9c0e7c58d296167fb8"}
Jul 28 18:45:54.045894 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.045871    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:45:54.110235 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls\") pod \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") "
Jul 28 18:45:54.110432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config\") pod \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") "
Jul 28 18:45:54.110432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59mvq\" (UniqueName: \"kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq\") pod \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") "
Jul 28 18:45:54.110432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110321    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location\") pod \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") "
Jul 28 18:45:54.110432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert\") pod \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\" (UID: \"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0\") "
Jul 28 18:45:54.110717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110681    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" (UID: "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:54.110717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110686    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" (UID: "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:45:54.110899 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.110769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" (UID: "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:45:54.112272 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.112251    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" (UID: "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:45:54.112371 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.112293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq" (OuterVolumeSpecName: "kube-api-access-59mvq") pod "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" (UID: "51ea1906-fe1b-41a9-bcb3-7e3aa22486e0"). InnerVolumeSpecName "kube-api-access-59mvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:45:54.211605 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.211584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:45:54.211691 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.211607    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59mvq\" (UniqueName: \"kubernetes.io/projected/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kube-api-access-59mvq\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:45:54.211691 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.211616    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:45:54.211691 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.211625    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:45:54.211691 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.211633    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:45:54.803832 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.803799    2561 generic.go:356] "Generic (PLEG): container finished" podID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerID="e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1" exitCode=0
Jul 28 18:45:54.804014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.803872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerDied","Data":"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"}
Jul 28 18:45:54.804014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.803877    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"
Jul 28 18:45:54.804014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.803918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55" event={"ID":"51ea1906-fe1b-41a9-bcb3-7e3aa22486e0","Type":"ContainerDied","Data":"4467d9a1c08e2c43948907bbf509c458e89ece52228091e625fdb2bab82fb3c9"}
Jul 28 18:45:54.804014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.803945    2561 scope.go:117] "RemoveContainer" containerID="aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"
Jul 28 18:45:54.811476 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.811461    2561 scope.go:117] "RemoveContainer" containerID="e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"
Jul 28 18:45:54.818367 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.818349    2561 scope.go:117] "RemoveContainer" containerID="14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"
Jul 28 18:45:54.823203 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.823179    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:45:54.825909 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.825892    2561 scope.go:117] "RemoveContainer" containerID="aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"
Jul 28 18:45:54.826183 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:45:54.826165    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24\": container with ID starting with aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24 not found: ID does not exist" containerID="aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"
Jul 28 18:45:54.826264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.826189    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24"} err="failed to get container status \"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24\": rpc error: code = NotFound desc = could not find container \"aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24\": container with ID starting with aceae1214135c7f2f2ebc69b289422be34425d2e06305acff796acbeb950ae24 not found: ID does not exist"
Jul 28 18:45:54.826264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.826207    2561 scope.go:117] "RemoveContainer" containerID="e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"
Jul 28 18:45:54.826520 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:45:54.826504    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1\": container with ID starting with e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1 not found: ID does not exist" containerID="e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"
Jul 28 18:45:54.826582 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.826524    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1"} err="failed to get container status \"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1\": rpc error: code = NotFound desc = could not find container \"e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1\": container with ID starting with e9793001ef9d3301c2c9f67c27f0f5504cef757fac5f973a263fbe4d39011da1 not found: ID does not exist"
Jul 28 18:45:54.826582 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.826538    2561 scope.go:117] "RemoveContainer" containerID="14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"
Jul 28 18:45:54.826772 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:45:54.826751    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1\": container with ID starting with 14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1 not found: ID does not exist" containerID="14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"
Jul 28 18:45:54.826829 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.826778    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1"} err="failed to get container status \"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1\": rpc error: code = NotFound desc = could not find container \"14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1\": container with ID starting with 14756b9aad543633655e6f90c8ff197c9c60bc72c3e913586ff13324171793e1 not found: ID does not exist"
Jul 28 18:45:54.827232 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:54.827212    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55"]
Jul 28 18:45:56.204289 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:56.204248    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" path="/var/lib/kubelet/pods/51ea1906-fe1b-41a9-bcb3-7e3aa22486e0/volumes"
Jul 28 18:45:56.813107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:56.813081    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/0.log"
Jul 28 18:45:56.813286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:56.813114    2561 generic.go:356] "Generic (PLEG): container finished" podID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerID="779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3" exitCode=1
Jul 28 18:45:56.813286 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:56.813192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerDied","Data":"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"}
Jul 28 18:45:57.818253 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:57.818225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/0.log"
Jul 28 18:45:57.818689 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:45:57.818346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerStarted","Data":"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"}
Jul 28 18:46:00.613621 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:00.613587    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:46:00.613986 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:00.613821    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer" containerID="cri-o://3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0" gracePeriod=30
Jul 28 18:46:01.701426 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701390    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="storage-initializer"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701693    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="storage-initializer"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701708    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701713    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701729    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kube-rbac-proxy"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701735    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kube-rbac-proxy"
Jul 28 18:46:01.701779 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701781    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kserve-container"
Jul 28 18:46:01.702008 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.701790    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ea1906-fe1b-41a9-bcb3-7e3aa22486e0" containerName="kube-rbac-proxy"
Jul 28 18:46:01.704804 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.704789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.727877 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.727855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 18:46:01.728199 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.728179    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:46:01.728199 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.728189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:46:01.730609 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.730589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:46:01.771955 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.771923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldgc\" (UniqueName: \"kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.771955 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.771954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.772131 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.771974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.772131 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.772050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.772131 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.772082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.872836 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.872802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldgc\" (UniqueName: \"kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.872836 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.872835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873072 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.872855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873072 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.872901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873072 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.872929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873072 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:46:01.873048    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 18:46:01.873291 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:46:01.873108    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls podName:e44bc1db-5c17-48f1-87d1-2fea76c46e5b nodeName:}" failed. No retries permitted until 2026-07-28 18:46:02.373092728 +0000 UTC m=+5330.690531605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 18:46:01.873397 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.873378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873538 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.873521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.873581 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.873566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:01.885107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:01.885083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldgc\" (UniqueName: \"kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:02.247813 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.247792    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/1.log"
Jul 28 18:46:02.248177 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.248159    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/0.log"
Jul 28 18:46:02.250494 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.250476    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:46:02.377581 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bgks\" (UniqueName: \"kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks\") pod \"679b596e-17b3-4ee1-89ec-8f1466f45533\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") "
Jul 28 18:46:02.377581 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config\") pod \"679b596e-17b3-4ee1-89ec-8f1466f45533\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") "
Jul 28 18:46:02.377581 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls\") pod \"679b596e-17b3-4ee1-89ec-8f1466f45533\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") "
Jul 28 18:46:02.377832 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location\") pod \"679b596e-17b3-4ee1-89ec-8f1466f45533\" (UID: \"679b596e-17b3-4ee1-89ec-8f1466f45533\") "
Jul 28 18:46:02.377832 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:02.377946 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "679b596e-17b3-4ee1-89ec-8f1466f45533" (UID: "679b596e-17b3-4ee1-89ec-8f1466f45533"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:46:02.377989 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.377954    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "679b596e-17b3-4ee1-89ec-8f1466f45533" (UID: "679b596e-17b3-4ee1-89ec-8f1466f45533"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:46:02.379697 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.379674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "679b596e-17b3-4ee1-89ec-8f1466f45533" (UID: "679b596e-17b3-4ee1-89ec-8f1466f45533"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:46:02.379806 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.379733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks" (OuterVolumeSpecName: "kube-api-access-7bgks") pod "679b596e-17b3-4ee1-89ec-8f1466f45533" (UID: "679b596e-17b3-4ee1-89ec-8f1466f45533"). InnerVolumeSpecName "kube-api-access-7bgks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:46:02.380045 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.380027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:02.478567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.478536    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b596e-17b3-4ee1-89ec-8f1466f45533-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:46:02.478567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.478560    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgks\" (UniqueName: \"kubernetes.io/projected/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-api-access-7bgks\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:46:02.478567 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.478569    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b596e-17b3-4ee1-89ec-8f1466f45533-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:46:02.478796 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.478579    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b596e-17b3-4ee1-89ec-8f1466f45533-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:46:02.615620 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.615574    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:02.742913 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.742836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:46:02.745814 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:46:02.745784    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44bc1db_5c17_48f1_87d1_2fea76c46e5b.slice/crio-220f159545fd0bf659c1c36baf26db14da13475e8dcb82cc98b645403669eafe WatchSource:0}: Error finding container 220f159545fd0bf659c1c36baf26db14da13475e8dcb82cc98b645403669eafe: Status 404 returned error can't find the container with id 220f159545fd0bf659c1c36baf26db14da13475e8dcb82cc98b645403669eafe
Jul 28 18:46:02.840405 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840378    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/1.log"
Jul 28 18:46:02.840756 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2_679b596e-17b3-4ee1-89ec-8f1466f45533/storage-initializer/0.log"
Jul 28 18:46:02.840817 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840773    2561 generic.go:356] "Generic (PLEG): container finished" podID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerID="3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0" exitCode=1
Jul 28 18:46:02.840874 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerDied","Data":"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"}
Jul 28 18:46:02.840874 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840856    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"
Jul 28 18:46:02.840981 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840875    2561 scope.go:117] "RemoveContainer" containerID="3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"
Jul 28 18:46:02.840981 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.840862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2" event={"ID":"679b596e-17b3-4ee1-89ec-8f1466f45533","Type":"ContainerDied","Data":"66a98e8b74711a4c66d9978aa74e7fad73cb0a80faff8e9c0e7c58d296167fb8"}
Jul 28 18:46:02.842730 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.842703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerStarted","Data":"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"}
Jul 28 18:46:02.842826 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.842752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerStarted","Data":"220f159545fd0bf659c1c36baf26db14da13475e8dcb82cc98b645403669eafe"}
Jul 28 18:46:02.850461 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.850441    2561 scope.go:117] "RemoveContainer" containerID="779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"
Jul 28 18:46:02.857701 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.857685    2561 scope.go:117] "RemoveContainer" containerID="3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"
Jul 28 18:46:02.857965 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:46:02.857947    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0\": container with ID starting with 3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0 not found: ID does not exist" containerID="3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"
Jul 28 18:46:02.858036 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.857978    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0"} err="failed to get container status \"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0\": rpc error: code = NotFound desc = could not find container \"3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0\": container with ID starting with 3867d1db6df2d4af4ae85258c363c42ca95073dace4f178a7564eab2cc8b70d0 not found: ID does not exist"
Jul 28 18:46:02.858036 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.858001    2561 scope.go:117] "RemoveContainer" containerID="779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"
Jul 28 18:46:02.858254 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:46:02.858238    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3\": container with ID starting with 779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3 not found: ID does not exist" containerID="779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"
Jul 28 18:46:02.858322 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.858259    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3"} err="failed to get container status \"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3\": rpc error: code = NotFound desc = could not find container \"779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3\": container with ID starting with 779874303eeffbdee523d3d62e7d2717939e886523c90227d718f992b67f3fb3 not found: ID does not exist"
Jul 28 18:46:02.896970 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.896941    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:46:02.900295 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:02.900269    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2"]
Jul 28 18:46:03.848495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:03.848465    2561 generic.go:356] "Generic (PLEG): container finished" podID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerID="97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0" exitCode=0
Jul 28 18:46:03.848850 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:03.848528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerDied","Data":"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"}
Jul 28 18:46:04.203048 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.203014    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" path="/var/lib/kubelet/pods/679b596e-17b3-4ee1-89ec-8f1466f45533/volumes"
Jul 28 18:46:04.854274 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.854234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerStarted","Data":"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"}
Jul 28 18:46:04.854778 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.854320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerStarted","Data":"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"}
Jul 28 18:46:04.854778 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.854377    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:04.854778 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.854411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:04.855755 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.855733    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:04.881618 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:04.881572    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podStartSLOduration=2.946941296 podStartE2EDuration="3.881555932s" totalImagesPullingTime="0s" totalInitContainerRuntime="934.614636ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:46:04.880388932 +0000 UTC m=+5333.197827818" watchObservedRunningTime="2026-07-28 18:46:04.881555932 +0000 UTC m=+5333.198994817"
Jul 28 18:46:05.858398 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:05.858360    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:10.862721 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:10.862692    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:46:10.863409 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:10.863377    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:20.864140 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:20.864097    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:30.863701 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:30.863644    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:40.864229 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:40.864184    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:46:50.863322 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:46:50.863277    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:47:00.864153 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:00.864088    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:47:10.863770 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:10.863733    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:47:20.864471 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:20.864438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:47:21.739098 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:21.739064    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:47:21.739586 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:21.739553    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container" containerID="cri-o://01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9" gracePeriod=30
Jul 28 18:47:21.739717 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:21.739655    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kube-rbac-proxy" containerID="cri-o://7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b" gracePeriod=30
Jul 28 18:47:22.159104 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.159073    2561 generic.go:356] "Generic (PLEG): container finished" podID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerID="7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b" exitCode=2
Jul 28 18:47:22.159494 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.159149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerDied","Data":"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"}
Jul 28 18:47:22.826418 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826382    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:22.826797 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826768    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.826797 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826788    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.826941 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826826    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.826941 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826837    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.826941 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826915    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.826941 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.826929    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="679b596e-17b3-4ee1-89ec-8f1466f45533" containerName="storage-initializer"
Jul 28 18:47:22.830174 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.830150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:22.833506 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.833484    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 18:47:22.833658 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.833568    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:47:22.842173 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.842144    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:22.957596 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.957556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx59p\" (UniqueName: \"kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:22.957782 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.957615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:22.957782 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.957641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:22.957782 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:22.957693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058358 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx59p\" (UniqueName: \"kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058358 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058358 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058502 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:23.058362    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 18:47:23.058502 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:23.058436    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls podName:27a036ee-1eb6-479c-b69a-80aec8f1312e nodeName:}" failed. No retries permitted until 2026-07-28 18:47:23.558416704 +0000 UTC m=+5411.875855587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" (UID: "27a036ee-1eb6-479c-b69a-80aec8f1312e") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 28 18:47:23.058612 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.058920 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.058899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.070627 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.070594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx59p\" (UniqueName: \"kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.562659 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.562620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.565063 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.565037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.740221 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.740186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:23.883424 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:23.883389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:23.886054 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:47:23.886024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a036ee_1eb6_479c_b69a_80aec8f1312e.slice/crio-b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53 WatchSource:0}: Error finding container b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53: Status 404 returned error can't find the container with id b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53
Jul 28 18:47:24.170391 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:24.170350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerStarted","Data":"3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"}
Jul 28 18:47:24.170568 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:24.170413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerStarted","Data":"b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53"}
Jul 28 18:47:25.858830 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:25.858779    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Jul 28 18:47:26.179630 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.179604    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/0.log"
Jul 28 18:47:26.179798 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.179640    2561 generic.go:356] "Generic (PLEG): container finished" podID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerID="3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab" exitCode=1
Jul 28 18:47:26.179798 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.179721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerDied","Data":"3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"}
Jul 28 18:47:26.896912 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.896882    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:47:26.993497 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config\") pod \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") "
Jul 28 18:47:26.993497 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert\") pod \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") "
Jul 28 18:47:26.993497 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993454    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location\") pod \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") "
Jul 28 18:47:26.993497 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ldgc\" (UniqueName: \"kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc\") pod \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") "
Jul 28 18:47:26.993855 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") pod \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\" (UID: \"e44bc1db-5c17-48f1-87d1-2fea76c46e5b\") "
Jul 28 18:47:26.993855 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e44bc1db-5c17-48f1-87d1-2fea76c46e5b" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:47:26.993855 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e44bc1db-5c17-48f1-87d1-2fea76c46e5b" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:47:26.993968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.993897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e44bc1db-5c17-48f1-87d1-2fea76c46e5b" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:47:26.995722 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.995699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e44bc1db-5c17-48f1-87d1-2fea76c46e5b" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:47:26.995802 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:26.995768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc" (OuterVolumeSpecName: "kube-api-access-8ldgc") pod "e44bc1db-5c17-48f1-87d1-2fea76c46e5b" (UID: "e44bc1db-5c17-48f1-87d1-2fea76c46e5b"). InnerVolumeSpecName "kube-api-access-8ldgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:47:27.095122 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.095078    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:27.095122 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.095114    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-cabundle-cert\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:27.095122 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.095127    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:27.095410 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.095140    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldgc\" (UniqueName: \"kubernetes.io/projected/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-kube-api-access-8ldgc\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:27.095410 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.095150    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bc1db-5c17-48f1-87d1-2fea76c46e5b-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:27.186060 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.186031    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/0.log"
Jul 28 18:47:27.186240 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.186147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerStarted","Data":"249bb1d1b19b2c10e8a87c7a5d1b5e499c961e971e84f38402e75bd4fe3fb056"}
Jul 28 18:47:27.188132 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.188107    2561 generic.go:356] "Generic (PLEG): container finished" podID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerID="01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9" exitCode=0
Jul 28 18:47:27.188266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.188149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerDied","Data":"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"}
Jul 28 18:47:27.188266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.188192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7" event={"ID":"e44bc1db-5c17-48f1-87d1-2fea76c46e5b","Type":"ContainerDied","Data":"220f159545fd0bf659c1c36baf26db14da13475e8dcb82cc98b645403669eafe"}
Jul 28 18:47:27.188266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.188218    2561 scope.go:117] "RemoveContainer" containerID="7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"
Jul 28 18:47:27.188266 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.188153    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"
Jul 28 18:47:27.196577 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.196553    2561 scope.go:117] "RemoveContainer" containerID="01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"
Jul 28 18:47:27.204644 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.204623    2561 scope.go:117] "RemoveContainer" containerID="97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"
Jul 28 18:47:27.212581 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.212562    2561 scope.go:117] "RemoveContainer" containerID="7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"
Jul 28 18:47:27.212820 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:27.212802    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b\": container with ID starting with 7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b not found: ID does not exist" containerID="7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"
Jul 28 18:47:27.212863 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.212830    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b"} err="failed to get container status \"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b\": rpc error: code = NotFound desc = could not find container \"7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b\": container with ID starting with 7af9377c4244710de1ba29e3b52e10870fcc5c2fecf1b4ae30dbba71a889f83b not found: ID does not exist"
Jul 28 18:47:27.212863 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.212848    2561 scope.go:117] "RemoveContainer" containerID="01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"
Jul 28 18:47:27.213065 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:27.213052    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9\": container with ID starting with 01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9 not found: ID does not exist" containerID="01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"
Jul 28 18:47:27.213107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.213070    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9"} err="failed to get container status \"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9\": rpc error: code = NotFound desc = could not find container \"01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9\": container with ID starting with 01675ac05067d155f93eff780138f394ce7cdc0449a70998ce2f4e8f0f920ee9 not found: ID does not exist"
Jul 28 18:47:27.213107 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.213082    2561 scope.go:117] "RemoveContainer" containerID="97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"
Jul 28 18:47:27.213274 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:27.213258    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0\": container with ID starting with 97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0 not found: ID does not exist" containerID="97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"
Jul 28 18:47:27.213353 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.213279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0"} err="failed to get container status \"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0\": rpc error: code = NotFound desc = could not find container \"97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0\": container with ID starting with 97c8e816b2b222197512682c21993550a9d55ce378bd25b4ec5e17e6b8805bc0 not found: ID does not exist"
Jul 28 18:47:27.218530 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.218505    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:47:27.222202 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:27.222177    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7"]
Jul 28 18:47:28.202998 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:28.202964    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" path="/var/lib/kubelet/pods/e44bc1db-5c17-48f1-87d1-2fea76c46e5b/volumes"
Jul 28 18:47:32.216889 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.216860    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/1.log"
Jul 28 18:47:32.217350 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.217196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/0.log"
Jul 28 18:47:32.217350 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.217230    2561 generic.go:356] "Generic (PLEG): container finished" podID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerID="249bb1d1b19b2c10e8a87c7a5d1b5e499c961e971e84f38402e75bd4fe3fb056" exitCode=1
Jul 28 18:47:32.217350 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.217266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerDied","Data":"249bb1d1b19b2c10e8a87c7a5d1b5e499c961e971e84f38402e75bd4fe3fb056"}
Jul 28 18:47:32.217350 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.217346    2561 scope.go:117] "RemoveContainer" containerID="3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"
Jul 28 18:47:32.217695 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.217675    2561 scope.go:117] "RemoveContainer" containerID="3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"
Jul 28 18:47:32.229633 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:32.229600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_kserve-ci-e2e-test(27a036ee-1eb6-479c-b69a-80aec8f1312e)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e"
Jul 28 18:47:32.229991 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:47:32.229961    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_kserve-ci-e2e-test_27a036ee-1eb6-479c-b69a-80aec8f1312e_0 in pod sandbox b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53 from index: no such id: '3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab'" containerID="3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"
Jul 28 18:47:32.230090 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.229996    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_kserve-ci-e2e-test_27a036ee-1eb6-479c-b69a-80aec8f1312e_0 in pod sandbox b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53 from index: no such id: '3e4bcd7efe8d27ccbc0703aa3172af59bd7de05098d9a4d3ccce57ba2351c3ab'"
Jul 28 18:47:32.813586 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:32.813550    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:33.222216 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.222187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/1.log"
Jul 28 18:47:33.351800 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.351779    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/1.log"
Jul 28 18:47:33.355318 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.355288    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:33.447810 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.447781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx59p\" (UniqueName: \"kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p\") pod \"27a036ee-1eb6-479c-b69a-80aec8f1312e\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") "
Jul 28 18:47:33.447998 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.447842    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config\") pod \"27a036ee-1eb6-479c-b69a-80aec8f1312e\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") "
Jul 28 18:47:33.447998 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.447874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location\") pod \"27a036ee-1eb6-479c-b69a-80aec8f1312e\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") "
Jul 28 18:47:33.447998 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.447914    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") pod \"27a036ee-1eb6-479c-b69a-80aec8f1312e\" (UID: \"27a036ee-1eb6-479c-b69a-80aec8f1312e\") "
Jul 28 18:47:33.448236 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.448198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27a036ee-1eb6-479c-b69a-80aec8f1312e" (UID: "27a036ee-1eb6-479c-b69a-80aec8f1312e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:47:33.448365 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.448297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27a036ee-1eb6-479c-b69a-80aec8f1312e" (UID: "27a036ee-1eb6-479c-b69a-80aec8f1312e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:47:33.449830 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.449801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p" (OuterVolumeSpecName: "kube-api-access-fx59p") pod "27a036ee-1eb6-479c-b69a-80aec8f1312e" (UID: "27a036ee-1eb6-479c-b69a-80aec8f1312e"). InnerVolumeSpecName "kube-api-access-fx59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:47:33.449907 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.449872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27a036ee-1eb6-479c-b69a-80aec8f1312e" (UID: "27a036ee-1eb6-479c-b69a-80aec8f1312e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:47:33.548851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.548771    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx59p\" (UniqueName: \"kubernetes.io/projected/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-api-access-fx59p\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:33.548851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.548795    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27a036ee-1eb6-479c-b69a-80aec8f1312e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:33.548851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.548804    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a036ee-1eb6-479c-b69a-80aec8f1312e-kserve-provision-location\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:33.548851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:33.548815    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27a036ee-1eb6-479c-b69a-80aec8f1312e-proxy-tls\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:47:34.228451 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.228425    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_27a036ee-1eb6-479c-b69a-80aec8f1312e/storage-initializer/1.log"
Jul 28 18:47:34.228835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.228524    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"
Jul 28 18:47:34.228835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.228550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf" event={"ID":"27a036ee-1eb6-479c-b69a-80aec8f1312e","Type":"ContainerDied","Data":"b771ec5c782bb38622f4d362451a51dc94760d4284b513b863f5e5579c761e53"}
Jul 28 18:47:34.228835 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.228618    2561 scope.go:117] "RemoveContainer" containerID="249bb1d1b19b2c10e8a87c7a5d1b5e499c961e971e84f38402e75bd4fe3fb056"
Jul 28 18:47:34.264328 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.264281    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:34.268540 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:34.268512    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf"]
Jul 28 18:47:35.258633 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258600    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qj5cs/must-gather-ctgj6"]
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258891    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258921    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258927    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258935    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kube-rbac-proxy"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258941    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kube-rbac-proxy"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258949    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="storage-initializer"
Jul 28 18:47:35.258997 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.258954    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="storage-initializer"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259003    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kserve-container"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259015    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259022    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44bc1db-5c17-48f1-87d1-2fea76c46e5b" containerName="kube-rbac-proxy"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259027    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259069    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.259264 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.259075    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" containerName="storage-initializer"
Jul 28 18:47:35.263540 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.263518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.265929 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.265905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qj5cs\"/\"openshift-service-ca.crt\""
Jul 28 18:47:35.266061 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.266045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qj5cs\"/\"kube-root-ca.crt\""
Jul 28 18:47:35.273217 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.273192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qj5cs/must-gather-ctgj6"]
Jul 28 18:47:35.366722 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.366685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.366722 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.366725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njl65\" (UniqueName: \"kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.468085 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.468045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.468085 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.468084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njl65\" (UniqueName: \"kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.468489 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.468467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.476268 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.476244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njl65\" (UniqueName: \"kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65\") pod \"must-gather-ctgj6\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") " pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.583640 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.583555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:47:35.711727 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.711700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qj5cs/must-gather-ctgj6"]
Jul 28 18:47:35.714032 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:47:35.714008    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73370e96_a440_4606_86c9_577b471a53e7.slice/crio-8f76b1628a45a5bbc5768af325ab2674218d9e72609fe63e8b3f6eb57b556554 WatchSource:0}: Error finding container 8f76b1628a45a5bbc5768af325ab2674218d9e72609fe63e8b3f6eb57b556554: Status 404 returned error can't find the container with id 8f76b1628a45a5bbc5768af325ab2674218d9e72609fe63e8b3f6eb57b556554
Jul 28 18:47:35.715658 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:35.715643    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:47:36.203614 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:36.203577    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a036ee-1eb6-479c-b69a-80aec8f1312e" path="/var/lib/kubelet/pods/27a036ee-1eb6-479c-b69a-80aec8f1312e/volumes"
Jul 28 18:47:36.238113 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:36.238075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" event={"ID":"73370e96-a440-4606-86c9-577b471a53e7","Type":"ContainerStarted","Data":"8f76b1628a45a5bbc5768af325ab2674218d9e72609fe63e8b3f6eb57b556554"}
Jul 28 18:47:42.270235 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:42.270195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" event={"ID":"73370e96-a440-4606-86c9-577b471a53e7","Type":"ContainerStarted","Data":"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"}
Jul 28 18:47:43.282359 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:43.282301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" event={"ID":"73370e96-a440-4606-86c9-577b471a53e7","Type":"ContainerStarted","Data":"d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a"}
Jul 28 18:47:43.313654 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:47:43.313596    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" podStartSLOduration=1.928043481 podStartE2EDuration="8.313579573s" totalImagesPullingTime="6.385536092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:47:43.311985915 +0000 UTC m=+5431.629424803" watchObservedRunningTime="2026-07-28 18:47:43.313579573 +0000 UTC m=+5431.631018460"
Jul 28 18:48:09.400530 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:09.400494    2561 generic.go:356] "Generic (PLEG): container finished" podID="73370e96-a440-4606-86c9-577b471a53e7" containerID="12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1" exitCode=0
Jul 28 18:48:09.400954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:09.400569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" event={"ID":"73370e96-a440-4606-86c9-577b471a53e7","Type":"ContainerDied","Data":"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"}
Jul 28 18:48:09.400954 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:09.400902    2561 scope.go:117] "RemoveContainer" containerID="12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"
Jul 28 18:48:10.189252 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:10.189219    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qj5cs_must-gather-ctgj6_73370e96-a440-4606-86c9-577b471a53e7/gather/0.log"
Jul 28 18:48:11.103566 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.103528    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2g68f/must-gather-5vrck"]
Jul 28 18:48:11.107260 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.107238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.110086 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.110059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2g68f\"/\"kube-root-ca.crt\""
Jul 28 18:48:11.111014 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.110997    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2g68f\"/\"default-dockercfg-mjh7z\""
Jul 28 18:48:11.111124 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.110997    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2g68f\"/\"openshift-service-ca.crt\""
Jul 28 18:48:11.117028 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.117002    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/must-gather-5vrck"]
Jul 28 18:48:11.181430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.181395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b3e0b29-2017-43b3-bc37-61e20948384d-must-gather-output\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.181615 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.181463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtd4\" (UniqueName: \"kubernetes.io/projected/0b3e0b29-2017-43b3-bc37-61e20948384d-kube-api-access-pbtd4\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.281914 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.281877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtd4\" (UniqueName: \"kubernetes.io/projected/0b3e0b29-2017-43b3-bc37-61e20948384d-kube-api-access-pbtd4\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.282116 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.281929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b3e0b29-2017-43b3-bc37-61e20948384d-must-gather-output\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.282244 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.282220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b3e0b29-2017-43b3-bc37-61e20948384d-must-gather-output\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.290875 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.290849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtd4\" (UniqueName: \"kubernetes.io/projected/0b3e0b29-2017-43b3-bc37-61e20948384d-kube-api-access-pbtd4\") pod \"must-gather-5vrck\" (UID: \"0b3e0b29-2017-43b3-bc37-61e20948384d\") " pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.416879 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.416842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/must-gather-5vrck"
Jul 28 18:48:11.552943 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:11.552675    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/must-gather-5vrck"]
Jul 28 18:48:11.555353 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:48:11.555324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3e0b29_2017_43b3_bc37_61e20948384d.slice/crio-c85127b257e081584de5b9afd250d823e18ee065a9e4a6378504dd3b0fdbf959 WatchSource:0}: Error finding container c85127b257e081584de5b9afd250d823e18ee065a9e4a6378504dd3b0fdbf959: Status 404 returned error can't find the container with id c85127b257e081584de5b9afd250d823e18ee065a9e4a6378504dd3b0fdbf959
Jul 28 18:48:12.414729 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:12.414694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/must-gather-5vrck" event={"ID":"0b3e0b29-2017-43b3-bc37-61e20948384d","Type":"ContainerStarted","Data":"c85127b257e081584de5b9afd250d823e18ee065a9e4a6378504dd3b0fdbf959"}
Jul 28 18:48:13.424869 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:13.424222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/must-gather-5vrck" event={"ID":"0b3e0b29-2017-43b3-bc37-61e20948384d","Type":"ContainerStarted","Data":"033e5032f369d56ade701c8c4a46dbef0c414c4a3018539f3d73d48fc55c0642"}
Jul 28 18:48:13.424869 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:13.424281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/must-gather-5vrck" event={"ID":"0b3e0b29-2017-43b3-bc37-61e20948384d","Type":"ContainerStarted","Data":"336a5ccbd9b3c320f719c11eecee2e0511e30965a668d8381cea2f27263bcdcd"}
Jul 28 18:48:13.442464 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:13.442404    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2g68f/must-gather-5vrck" podStartSLOduration=1.544885888 podStartE2EDuration="2.442383004s" totalImagesPullingTime="897.497116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:48:13.440571221 +0000 UTC m=+5461.758010131" watchObservedRunningTime="2026-07-28 18:48:13.442383004 +0000 UTC m=+5461.759821895"
Jul 28 18:48:14.028460 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:14.028428    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rw72f_fa356ac2-7704-422c-adac-5885e9238409/global-pull-secret-syncer/0.log"
Jul 28 18:48:14.169201 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:14.169169    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rcqhx_9edf578f-d23b-45c5-81b6-582b69a24d5f/konnectivity-agent/0.log"
Jul 28 18:48:14.221526 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:14.221480    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-92.ec2.internal_e04dce7c55a72020d01171561fa4ac33/haproxy/0.log"
Jul 28 18:48:15.946092 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:15.946046    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-qj5cs/must-gather-ctgj6"]
Jul 28 18:48:15.946628 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:15.946294    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-qj5cs/must-gather-ctgj6" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="copy" containerID="cri-o://d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a" gracePeriod=2
Jul 28 18:48:15.954083 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:15.954043    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-qj5cs/must-gather-ctgj6"]
Jul 28 18:48:16.327326 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.325751    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qj5cs_must-gather-ctgj6_73370e96-a440-4606-86c9-577b471a53e7/copy/0.log"
Jul 28 18:48:16.331715 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.331607    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:48:16.433646 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.433603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output\") pod \"73370e96-a440-4606-86c9-577b471a53e7\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") "
Jul 28 18:48:16.433959 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.433707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njl65\" (UniqueName: \"kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65\") pod \"73370e96-a440-4606-86c9-577b471a53e7\" (UID: \"73370e96-a440-4606-86c9-577b471a53e7\") "
Jul 28 18:48:16.437336 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.437155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65" (OuterVolumeSpecName: "kube-api-access-njl65") pod "73370e96-a440-4606-86c9-577b471a53e7" (UID: "73370e96-a440-4606-86c9-577b471a53e7"). InnerVolumeSpecName "kube-api-access-njl65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:48:16.439316 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.439265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "73370e96-a440-4606-86c9-577b471a53e7" (UID: "73370e96-a440-4606-86c9-577b471a53e7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:16.440070 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.440044    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qj5cs_must-gather-ctgj6_73370e96-a440-4606-86c9-577b471a53e7/copy/0.log"
Jul 28 18:48:16.440803 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.440767    2561 generic.go:356] "Generic (PLEG): container finished" podID="73370e96-a440-4606-86c9-577b471a53e7" containerID="d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a" exitCode=143
Jul 28 18:48:16.440899 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.440870    2561 scope.go:117] "RemoveContainer" containerID="d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a"
Jul 28 18:48:16.441137 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.441053    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qj5cs/must-gather-ctgj6"
Jul 28 18:48:16.465799 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.465498    2561 scope.go:117] "RemoveContainer" containerID="12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.500153    2561 scope.go:117] "RemoveContainer" containerID="d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:48:16.501965    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a\": container with ID starting with d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a not found: ID does not exist" containerID="d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.502001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a"} err="failed to get container status \"d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a\": rpc error: code = NotFound desc = could not find container \"d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a\": container with ID starting with d988b09853a78d20534a90c8a28af70ecbd8898dbc6c3d936214f680bbc47d1a not found: ID does not exist"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.502038    2561 scope.go:117] "RemoveContainer" containerID="12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: E0728 18:48:16.502375    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1\": container with ID starting with 12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1 not found: ID does not exist" containerID="12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"
Jul 28 18:48:16.502794 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.502417    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1"} err="failed to get container status \"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1\": rpc error: code = NotFound desc = could not find container \"12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1\": container with ID starting with 12d625c1d796ee548ee8094257783ad7505de9108b78ad0191bd11535e3e55a1 not found: ID does not exist"
Jul 28 18:48:16.535172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.535124    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njl65\" (UniqueName: \"kubernetes.io/projected/73370e96-a440-4606-86c9-577b471a53e7-kube-api-access-njl65\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:48:16.535172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:16.535173    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73370e96-a440-4606-86c9-577b471a53e7-must-gather-output\") on node \"ip-10-0-138-92.ec2.internal\" DevicePath \"\""
Jul 28 18:48:18.206867 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:18.206820    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73370e96-a440-4606-86c9-577b471a53e7" path="/var/lib/kubelet/pods/73370e96-a440-4606-86c9-577b471a53e7/volumes"
Jul 28 18:48:19.027532 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.027501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-848dc66969-kr6xw_cdb44e09-af52-4f56-8b19-c663120358b3/metrics-server/0.log"
Jul 28 18:48:19.064585 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.064546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-ggjbt_99794383-ea28-46e3-8b5e-f6e31f485c08/monitoring-plugin/0.log"
Jul 28 18:48:19.099378 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.099347    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77b8l_99e87093-78bc-46a5-86af-7b8292f8bf38/node-exporter/0.log"
Jul 28 18:48:19.123330 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.123287    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77b8l_99e87093-78bc-46a5-86af-7b8292f8bf38/kube-rbac-proxy/0.log"
Jul 28 18:48:19.147479 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.147444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77b8l_99e87093-78bc-46a5-86af-7b8292f8bf38/init-textfile/0.log"
Jul 28 18:48:19.474317 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.474217    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/prometheus/0.log"
Jul 28 18:48:19.497741 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.497698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/config-reloader/0.log"
Jul 28 18:48:19.533517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.533460    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/thanos-sidecar/0.log"
Jul 28 18:48:19.560558 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.560525    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/kube-rbac-proxy-web/0.log"
Jul 28 18:48:19.583738 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.583711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/kube-rbac-proxy/0.log"
Jul 28 18:48:19.614554 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.614520    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/kube-rbac-proxy-thanos/0.log"
Jul 28 18:48:19.638588 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.638560    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6c0df93a-7f35-4538-bbc2-19323866a040/init-config-reloader/0.log"
Jul 28 18:48:19.674431 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.674352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rrmhq_ecc30941-eb09-4145-8f16-c72d17284331/prometheus-operator/0.log"
Jul 28 18:48:19.698034 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.698009    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rrmhq_ecc30941-eb09-4145-8f16-c72d17284331/kube-rbac-proxy/0.log"
Jul 28 18:48:19.883069 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.883036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/thanos-query/0.log"
Jul 28 18:48:19.914545 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.914512    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/kube-rbac-proxy-web/0.log"
Jul 28 18:48:19.941495 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.941466    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/kube-rbac-proxy/0.log"
Jul 28 18:48:19.975746 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:19.975715    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/prom-label-proxy/0.log"
Jul 28 18:48:20.009263 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:20.009229    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/kube-rbac-proxy-rules/0.log"
Jul 28 18:48:20.044433 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:20.044398    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5475cdfdb9-8bt6b_7c67b470-8025-42d7-8c47-801a91bc15d9/kube-rbac-proxy-metrics/0.log"
Jul 28 18:48:22.176184 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.176150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bb94f5997-w2qrm_b7e0b011-8ae8-4c68-b14d-e2153ff229fa/console/0.log"
Jul 28 18:48:22.625172 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625082    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"]
Jul 28 18:48:22.625619 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="copy"
Jul 28 18:48:22.625619 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625621    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="copy"
Jul 28 18:48:22.625788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625660    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="gather"
Jul 28 18:48:22.625788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625668    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="gather"
Jul 28 18:48:22.625788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625746    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="gather"
Jul 28 18:48:22.625788 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.625762    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73370e96-a440-4606-86c9-577b471a53e7" containerName="copy"
Jul 28 18:48:22.631480 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.631454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.639432 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.638654    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"]
Jul 28 18:48:22.799530 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.799485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-sys\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.799719 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.799623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-proc\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.799719 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.799699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ljp\" (UniqueName: \"kubernetes.io/projected/97eb930c-0532-44b6-abe0-620b627f097c-kube-api-access-88ljp\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.799849 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.799730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-lib-modules\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.799849 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.799831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-podres\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-proc\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900606 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ljp\" (UniqueName: \"kubernetes.io/projected/97eb930c-0532-44b6-abe0-620b627f097c-kube-api-access-88ljp\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-lib-modules\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-proc\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-podres\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-sys\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-lib-modules\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-sys\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.900851 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.900830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97eb930c-0532-44b6-abe0-620b627f097c-podres\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.909945 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.909905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ljp\" (UniqueName: \"kubernetes.io/projected/97eb930c-0532-44b6-abe0-620b627f097c-kube-api-access-88ljp\") pod \"perf-node-gather-daemonset-9m7w5\" (UID: \"97eb930c-0532-44b6-abe0-620b627f097c\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:22.944228 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:22.944191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:23.109004 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.108964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"]
Jul 28 18:48:23.113587 ip-10-0-138-92 kubenswrapper[2561]: W0728 18:48:23.113550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97eb930c_0532_44b6_abe0_620b627f097c.slice/crio-5ba30dca9419745bcbae414a9004179c541082e0f9ce469a61329bb81ade6c55 WatchSource:0}: Error finding container 5ba30dca9419745bcbae414a9004179c541082e0f9ce469a61329bb81ade6c55: Status 404 returned error can't find the container with id 5ba30dca9419745bcbae414a9004179c541082e0f9ce469a61329bb81ade6c55
Jul 28 18:48:23.482093 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.482044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5" event={"ID":"97eb930c-0532-44b6-abe0-620b627f097c","Type":"ContainerStarted","Data":"ce58483f5fdbed896e9f47a7ec4c76736e24de77bafbd002f7a6ece6ee25355e"}
Jul 28 18:48:23.482714 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.482120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5" event={"ID":"97eb930c-0532-44b6-abe0-620b627f097c","Type":"ContainerStarted","Data":"5ba30dca9419745bcbae414a9004179c541082e0f9ce469a61329bb81ade6c55"}
Jul 28 18:48:23.482855 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.482830    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:23.498462 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.498400    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5" podStartSLOduration=1.498382619 podStartE2EDuration="1.498382619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:48:23.497737592 +0000 UTC m=+5471.815176479" watchObservedRunningTime="2026-07-28 18:48:23.498382619 +0000 UTC m=+5471.815821507"
Jul 28 18:48:23.766446 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.766374    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mh54r_e281de0b-41cf-4e5d-bb3f-263bd543b1f1/dns/0.log"
Jul 28 18:48:23.794374 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.794339    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mh54r_e281de0b-41cf-4e5d-bb3f-263bd543b1f1/kube-rbac-proxy/0.log"
Jul 28 18:48:23.873734 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:23.873696    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmvp4_ecb1f853-e9c4-4d6a-8540-f76de0162149/dns-node-resolver/0.log"
Jul 28 18:48:24.458918 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:24.458878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knslj_7bcaf0bd-eabc-4f4f-a938-83f807904b03/node-ca/0.log"
Jul 28 18:48:25.596058 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:25.596029    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7vqvq_be2708eb-224f-4dd7-bfa0-0fc62342849d/serve-healthcheck-canary/0.log"
Jul 28 18:48:26.064781 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:26.064752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdrjx_289b8896-51ca-4643-8767-f34ba8c2ab88/kube-rbac-proxy/0.log"
Jul 28 18:48:26.088069 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:26.088039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdrjx_289b8896-51ca-4643-8767-f34ba8c2ab88/exporter/0.log"
Jul 28 18:48:26.112843 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:26.112816    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gdrjx_289b8896-51ca-4643-8767-f34ba8c2ab88/extractor/0.log"
Jul 28 18:48:28.731615 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:28.731584    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-z7wgb_8cd4b27a-a718-4089-8a02-162bb1fda220/server/0.log"
Jul 28 18:48:29.065269 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.065178    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-vbjnd_cc2143ac-9761-4e73-a1bb-38d7d84ce1b0/manager/0.log"
Jul 28 18:48:29.096793 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.096764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qxz4t_80322afd-413f-4a62-b09b-e0f60938ae54/s3-init/0.log"
Jul 28 18:48:29.117354 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.117289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qxz4t_80322afd-413f-4a62-b09b-e0f60938ae54/download-hf-model/0.log"
Jul 28 18:48:29.148086 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.148053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-c8prc_18b17d9e-8e6d-49e2-8502-2300af7cf42c/s3-tls-init-custom/0.log"
Jul 28 18:48:29.168444 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.168409    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-c8prc_18b17d9e-8e6d-49e2-8502-2300af7cf42c/download-hf-model/0.log"
Jul 28 18:48:29.199856 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.199828    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4s49f_7a8a83f5-a79d-4594-8ee0-e2c7ea338977/s3-tls-init-serving/0.log"
Jul 28 18:48:29.218833 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.218802    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4s49f_7a8a83f5-a79d-4594-8ee0-e2c7ea338977/download-hf-model/0.log"
Jul 28 18:48:29.248504 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.248476    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-8mhbf_adca4fbe-b95e-4329-8b91-00cec369bf6d/seaweedfs/0.log"
Jul 28 18:48:29.273968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:29.273939    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-qqsn5_0509f459-d23e-4311-b7b9-805790d27b9b/seaweedfs-tls-custom/0.log"
Jul 28 18:48:30.500729 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:30.500704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-9m7w5"
Jul 28 18:48:35.011337 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.011289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2m8g7_6cadb470-ea44-4d68-ba72-673593f2f908/kube-multus/0.log"
Jul 28 18:48:35.038687 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.038655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:48:35.063901 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.063866    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/egress-router-binary-copy/0.log"
Jul 28 18:48:35.087998 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.087968    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/cni-plugins/0.log"
Jul 28 18:48:35.113854 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.113820    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/bond-cni-plugin/0.log"
Jul 28 18:48:35.138430 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.138393    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/routeoverride-cni/0.log"
Jul 28 18:48:35.162079 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.162053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/whereabouts-cni-bincopy/0.log"
Jul 28 18:48:35.185371 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.185332    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qw6b_5ef737e1-0408-47c9-962c-ac8ad414f763/whereabouts-cni/0.log"
Jul 28 18:48:35.681580 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.681555    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7nphk_13a33bf6-6706-40de-bdb3-a635e075db13/network-metrics-daemon/0.log"
Jul 28 18:48:35.702585 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:35.702557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7nphk_13a33bf6-6706-40de-bdb3-a635e075db13/kube-rbac-proxy/0.log"
Jul 28 18:48:36.554863 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.554834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/ovn-controller/0.log"
Jul 28 18:48:36.573484 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.573441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/ovn-acl-logging/0.log"
Jul 28 18:48:36.598162 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.598136    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/kube-rbac-proxy-node/0.log"
Jul 28 18:48:36.621078 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.621040    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:48:36.650517 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.650489    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/northd/0.log"
Jul 28 18:48:36.672968 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.672945    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/nbdb/0.log"
Jul 28 18:48:36.696812 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.696787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/sbdb/0.log"
Jul 28 18:48:36.750412 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:36.750381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6spk5_5befc373-0bba-4238-839c-7463d45b4913/ovnkube-controller/0.log"
Jul 28 18:48:38.229025 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:38.228992    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sck52_8e431ceb-b3e3-4d1a-a1b4-1cb71a4bbf92/network-check-target-container/0.log"
Jul 28 18:48:39.242005 ip-10-0-138-92 kubenswrapper[2561]: I0728 18:48:39.241976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vgb77_82531788-de8e-41d8-92ca-ec35a6e2521d/iptables-alerter/0.log"